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Foreword 

The National Cooperative Education Statistics System 

The work of the Forum is a key aspect of the National Cooperative Education Statistics System. The Cooperative 
System was established to produce and maintain, with the cooperation of the states, comparable and uniform education 
information and data that are useful for policymaking at the federal, state, and local levels. To assist in meeting this goal, 
the National Center for Education Statistics (NCES), within the U.S. Department of Education, established the Forum 
to improve the collection, reporting, and use of elementary and secondary education statistics. The Forum deals with 
issues in education data policy, sponsors innovations in data collection and reporting, and provides technical assistance to 
improve state and local data systems. 

Development of Forum Products 

Members of the Forum establish task forces to develop best practice guides in data-related areas of interest to federal, 
state, and local education agencies. They are assisted in this work by NCES, but the content comes from the collective 
experience of the state and school district task force members who review all products iteratively throughout the 
development process. Documents prepared, reviewed, and approved by task force members undergo a formal public 
review. This public review consists of focus groups comprised of representatives of the product’s intended audience, 
review sessions at relevant regional or national conferences, or technical reviews by acknowledged experts in the field. 
In addition, all draft documents are posted on the Forum website prior to publication so that any interested individuals 
or organizations can provide feedback. After the task force oversees the integration of public review comments 
and reviews the document a final time, publications are subject to examination by members of the Forum standing 
committee that is sponsoring the project. Finally, the entire Forum (approximately 1 20 members) reviews and formally 
votes to approve all documents prior to publication. NCES provides final review and approval prior to publication. 
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Overview 

This is the Introduction to a series of briefs intended to serve as a guide to the skillful and 
appropriate use of education data. The purpose of this resource is to provide readers with 
practical information about the knowledge, skills, and abilities needed to identify, access, 
interpret, and use data to improve instruction in classrooms and the operation of schools, 
local education agencies (LEAs, also known as school districts), and state education agencies 
(SEAs). 

Introduction: Expectations are Rising 

Countless pieces of data are collected each day about students and their educational 
experiences, including information about attendance, classroom participation, assessment 
results, grades, responses to test questions, and discipline referrals. Other data are collected 
about schools and districts, such as student enrollment, program participation, staff 
characteristics, and facility condition. Additional data are collected about states, including 
the number and types of schools, revenues and expenses, program availability, and teacher 
licensure. These data are used for a wide range of purposes and are frequently summarized, 
aggregated and disaggregated, and otherwise combined to describe the educational status of 
students, schools, school districts, and states. 

Data-driven decisionmaking is a term that was commonly referenced in the 1 990s — as 
if it was a new concept ushered in by the application of innovative technologies like 
spreadsheets, databases, electronic data exchange, and the more widespread use of 
computers in general. But, while powerful, data-driven decisionmaking is only part of the 
equation for school success. . . data-driven action is what really improves education. For 
example, determining that migrant children are underserved in rural schools is only part 
of an effective solution; data-driven action must follow to allocate funding for after-school 
programs that will address the service deficiency and better meet the educational needs of 
the students. 

The truth is that education professionals have been using actionable data for a long time, but 
in many cases these “data” were mostly observational or anecdotal in nature. For example, 
an astute teacher might recognize when there were a lot of misspelled words in a writing 
assignment, indicating that the class needed more spelling instruction; a good principal 
would discern when too many fights were occurring in a particularly narrow hallway, 
indicating that a hall monitor was needed or students should be rerouted; and a wise 
administrator would notice if fifth graders across the entire school district were missing the 
same types of questions on a standardized math test, suggesting that the curriculum might 
need to be modified to strengthen a particular unit. 


To be used effectively, 
data need to be 
accurate, accessible, 
understandable, timely, 
and actionable. 


All data use should 
occur within the context 
of an organization's 
data governance 
policies, which help 
to ensure proper 
oversight, high quality 
collection, suitable 
access, effective 
management, and 
appropriate use. 
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Not all data 
lead to useful 
information. 

Not every combination 
of data produces 
information that 
helps an education 
organization in a 
meaningful way. For 
example, an advocacy 
group in one state took 
advantage of readily 
available technology to 
cross tabulate all data 
published by its SEEA 
in order to create a 
volume of "indicators" 
for comparing the 
state's schools and 
districts. Because the 
organization included all 
available data elements 
and aggregates, 
the report included 
meaningless "indicators" 
such as Paper Towel 
Use per Student 
Mile Bused. 


Most data users don't 
need to become data 
experts, but they should 
possess the basic skills 
necessary to effectively 
use available data to 
make decisions and 
take actions related to 
their job responsibilities. 


In an era of diminishing public resources, the effective use of data is about being efficient: 
once an education organization has gone to the effort of collecting data, failing to use 
the information to inform instructional, administrative, and policy-related activities 
is uneconomical, unwise, and a waste of a valuable information resource. Conversely, 
appropriate action, based on the right data at the right time, can lead education 
organizations to greater efficiency, educators to greater effectiveness, and students to 
greater academic achievement. 


The Evolving Nature of Data Use 

Intuition: Putting experience to use in recognizing anecdotal data (e.g., observing 
students, schools, and programs, and using anecdotal evidence to characterize 
performance, as occurs when a teacher recalls that Johnny keeps misusing capital letters 
in his writing assignments and may need a review lesson). 

Compliance: Using data to determine whether an organization has met established 
requirements (e.g., reporting associated with funding streams such as special education 
or migrant education). 

Accountability: Reporting outcomes on any of a range of performance metrics for which 
an education agency is responsible (e.g., the academic proficiency of student groups). 

Decisionmaking: Relying on data to inform conclusions (e.g., instructional, operational, 
and policy choices). 

Evaluation: Using data to assess the results of choices and actions (e.g., relying on data 
to inform an evaluation of the effectiveness of a reading program). 

Data Use that Informs Action: Changing behavior based on what the data suggest (e.g., 
implementing interventions based on trends and outcomes identified through data)... and 
the goal of this document. 


Data Experts and Data Users 

Education organizations often employ professional staff who are skilled in the collection, 
management, and reporting of education data, but most data users are not data experts — 
nor do they need to be. Instead, data users need to have the knowledge, skills, and abilities 
necessary to identify, access, understand, analyze, interpret, and use education data as 
appropriate to perform their respective duties in a school, district office, or state education 
agency. Although the specific knowledge, skills, and abilities needed to be a proficient data 
user may vary by role and organization, all key stakeholders in the education enterprise 
should be able to effectively incorporate data into their decisionmaking and actions. 
Responsible data users understand both the meaning and limitations of data, especially in 
light of changing policies, regulations, and practices governing data collection. 

The Cycle of Data Use 

Data use, like any process, involves a series of steps that build upon one another to achieve 
a desired goal. With respect to education data, the goal is to take action to improve the 
education system and student learning. The recommendations in this document are based 


2 


Forum Guide to Taking Action with Education Data 




on a conceptual diagram of the cycle of data use that is intended to help readers appreciate 
the ways in which they will apply data use skills to accomplish their work in education 
organizations (figure l).This cycle of data use consists of five primary phases or activities: 
(1) seek information; (2) access/ gather data; (3) analyze/interpret data; (4) act; and (5) 
evaluate. Such a framework can serve as the basis for systematic data use for education 
stakeholders and will be revisited in each of the audience-specific briefs in this series. 


Figure 1. The Cycle of Data Use 

This cycle serves as the framework for organizing content in Briefs I, II. and III. 



4. Act 


Each step in the data 
use cycle supports 
the ultimate purpose 
of data use: taking 
action to improve the 
education system and 
student learning. 


Data Meaning: Even superficially similar terms can have very different meanings. For 
example 

• “class size" is not the same as “student/teacher ratio"; 

• “completion rate” is not the same as “graduation rate"; and 

• program “eligibility," “enrollment,” “participation," and “completion” are not 
identical terms. 

Data Limitations: There are logical, statistical, and common sense limits to the 
appropriate use of most types of data. For example 

• many datasets reflect counts on specific collection dates and are not necessarily 
representative of conditions at other times in the same school year; 

• private school students and homeschooled students are not necessarily included 
in many data collections and reports; and 

• policy choices can strongly affect data values— in some districts, for example, 
students may be counted as in attendance if they are present during homeroom, 
while other districts maintain attendance for each class period throughout the 
school day. 
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1. Seek Information 

Effective data users must refine their inquiries to meaningful and achievable units of 
analysis. They often begin by trying to recognize gaps between what they currently know 
and what they would like to know about their students, teachers, classrooms, curricula, 
programs, schools, districts, and states. Identifying these data “gaps” and subsequently 
trying to find information that fills those gaps leads a data user toward different sets of 
data depending on the information gap being addressed. As such, the design of a systematic 
data use process begins with key questions that support this most critical step in the wise 
and appropriate use of data for action: information seeking. Information seeking refers to 
the process of finding the right data to address the specific information needs at hand. For 
example, a very basic “information seeking” skill tells a teacher who is concerned about a 
sixth grade student’s multiplication skills to look at the child’s fifth grade math scores rather 
than the results of a recent spelling test. A more fine-tuned information seeker might know 
to look at the third quarter interim assessment results from the fifth grade, which focused 
on multiplication skills, rather than the summative assessment that covered a broad range 
of content standards. The point is that “information seeking” helps to appropriately fill 
the gap between what is currently known and what needs to be known by identifying the 
information (data) that most accurately, validly, and reliably provide the specific information 
needed to address the question(s) at hand. Steps in the information seeking process include 

"S recognizing when information is needed to inform a decision and direct 

an action; 

defining a meaningful and achievable scope of concern/ unit of analysis; 

clearly articulating the issue at hand as a critical question; 

S assessing what relevant data are already available; 

'k identifying “gaps” in currently available data; 

'k identifying potential barriers to finding the information and ways to overcome them; 
"S seeking additional data that are accurate, valid, reliable, timely, and relevant; and 
'k relying on data that are frequently valuable to education stakeholders including 

• enrollment data; 

• demographic data; 

• performance data; 

• engagement data; 

• health data; 

• college and career readiness data; 

• teacher data; 

• program data; 

• food and nutrition data; 

• financial data; 

• transportation data; 

• school data; 

• early childhood data; 

• college and university data; and 

• workforce data. 


4 


Forum Guide to Taking Action with Education Data 




2. Access/Gather Data 


Many people mistakenly believe that data are only collected in response to compliance 
and accountability mandates, but the reality is that data are collected for many purposes. 
Education data are collected to inform school operations, program management, resource 
allocation, and, not the least of all, instructional choices. In fact, many data collections are 
designed specifically to answer an organization’s real-world questions and information gaps 
(as described in step 1 above). As such, the role of a prospective data user is to gather data 
that are most relevant to the questions they wish to address. Gathering data might require 
a new data collection but, in many cases, it is often about accessing data that have already 
been collected by someone else. By actively searching for data that best relate to the key 
questions (information gaps) at hand, stakeholders can gain access to powerful information 
that will, when appropriately used, better inform the decisions they make while carrying 
out their educational responsibilities. Steps to accessing/ gathering data include 


•A identifying whether the needed data already exist; 

■A collecting the data, which often means gaining access to relevant data that have 
already been collected by someone else; and 

learning about the data, such as applicable definitions, limitations, timeliness, 
formatting, other data sets, etc., to ensure that the user fully understands the 
information (these “data about the data” are commonly referred to as metadata) . 

3. Analyze/interpret Data 

How a user analyzes data will depend largely on the questions that have been asked (Step 
1 : Seek Information), their access to data needed to answer the questions (Step 2: Access/ 
Gather Data), the analytical skills of the user, and the tools available to support analysis and 
interpretation (from paper reports to spreadsheets and software programs) . The goal of 
analysis and interpretation is to derive logically (or statistically) sound evidence to inform 
decisionmaking and action. Oftentimes, this includes answers to questions related to what 
is working and what’s not, what is changing over time, and what information is or is not 
available. Tasks related to analysis and interpretation include 

■A formatting the data in a manner that enables necessary analysis and 
interpretation; and 

■A determining the constraints of the data, such as the 

• unit of analysis (e.g. , individual or aggregate) 

• data design (e.g., random sampling, comparison of cohorts and peers, etc.) 

• timeframe (e.g. , a “snapshot” in time or longitudinal/ trend data) 

• data purposes (e.g., formative versus summative assessment) 

• data quality (e.g., accuracy, reliability, timeliness, and completeness) 

• data bias (e.g., is the source an advocacy group or an objective researcher?) 

• data misuse (e.g., are these data often confused with other types of data , 
such as average class size and student /teacher ratio?); and 

• producing logically (or statistically) sound and defensible conclusions. 


Data use to inform 
decisionmaking, not 
decisionmaking to 
inform data use. 

It may be tempting 
to make a decision 
and then search only 
for those data that 
support it. Such "cherry 
picking" of data is 
an inappropriate and 
unethical use of data 
and is inconsistent with 
the principle of sound 
data use. 
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The proof of skilled 
data use is the 
application of insights 
from data in a practical 
and actionable way to 
improve the education 
system. 



Figure 2. Analysis includes looking for patterns in the data. Rather than finding one 
piece of data and using it to make a generalization or draw conclusions that are beyond the 
scope of the single data point, identify multiple data sources that inform the analysis of the 
questions— a process sometimes referred to as “triangulation.” 


4. Act 


This document defines data use as the systematic process of identifying information needs, 
gathering data, and analyzing and interpreting information for the purpose of taking 
action to answer a question, address a problem, or change a situation. It should be noted, 
however, that the skilled use of education data is a necessary, but not sufficient, component 
of improving teaching and learning. Other important information resources include a 
stakeholder’s real-world experience, relevant laws and regulations, organizational policies 
and processes, and even theoretical research. Taken together, this body of information 
can help data users determine appropriate answers to their key questions, and enable 
the application of these insights in a practical and actionable way to improve educational 
processes. 

Once a stakeholder has an adequate understanding of data meaning and limitations (see 
Step 3: Analysis /Interpretation), ethics often play a key role in determining appropriate and 
inappropriate use. 1 For example, inappropriate uses of data include 


1 For more information about the ethical use of education data, see the Forum Guide to Data Ethics, available 
at http:/ / nces.ed.gov/Jorum/pub_20 10801 .asp. 
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* teachers using a single assessment score as the predominant component for 
evaluating a student; 

* principals using past summative assessment scores to exclude certain students 
from future testing; guidance counselors using demographic or socioeconomic 
data as the primary predictor of individual academic potential; and 

* state program administrators using geographic location as the sole criteria for 
awarding competitive grants to schools. 


Appropriate uses of the same data might include 


^ using a body of fair and relevant factors to evaluate students; 

^ using past summative assessment scores to identify students in need of 
supplemental instructional services prior to future testing; 
using demographic or socioeconomic data to encourage and embrace diversity in 
school settings and recognizing the unique potential of each individual student; and 
using a comprehensive set of indicators to determine the fair allocation of 
program funds to schools across the state. 


The cycle of data use 
presented in figure 1 
is iterative in nature, 
Evaluation leads to 
more information 
seeking and effectively 
starts the cycle over 
again. 


5. Evaluate 


The cycle of data use presented in figure 1 is iterative in nature. Acting on data in a 
responsible manner can change the status quo — and wise data users will want to evaluate 
whether such change improves the situation. Sometimes this type of evaluation generates 
new questions or data needs that arise in response to data-informed action. Effective 
evaluation efforts often address questions such as 

'E To what extent has the initial issue been resolved effectively? 

'E What new concerns have arisen? 

'd Which factors are understood sufficiently and which warrant 
additional investigation? 

'E Have new data gaps (needs) been identified? 

If the issue has been effectively addressed by data-informed action, it might be appropriate 
to share your experience with others and encourage them to use data to help with similar 
situations. If the issue hasn’t been fully resolved, the cycle of data use repeats itself. 

When this occurs, stakeholders may need to engage the data cycle process again from the 
beginning (Step 1 : Seek Information). 
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Skills Required to Effectively Analyze and Interpret Data 

In order for data to be used to inform action, education stakeholders must feel secure 
about their ability to access, analyze, and use data in classroom, school, district, and state 
agency settings. Developing this confidence requires that stakeholders know enough about 
the data to ascertain their quality and applicability. They must also have a command of 
basic analytical skills needed to properly interpret and attribute meaning to the data. This 
includes the following core knowledge, skills, and abilities: 

• understand what the data mean; 

• view the data in an appropriate context; 

• recognize the nature of the collection tool; 

• identify bias in the collection source; 

• assess the quality of the data; 

• appreciate the assumptions and limitations of the data; 

• comprehend the significance of error or uncertainty in the data; 

• discern meaning in tabular and graphical data; and 

• know how to effectively question data /report authors. 

s Understand what the data mean. “Metadata” refer to “data about 
data”’ and can include management items (e.g., definitions, formulas, usage 
restrictions, timeliness, and mandates), reporting information (e.g., anticipated 
use, key words, quality metrics), and technical information (e.g., field lengths, 
permitted values, translation history). High quality metadata deliver too much 
value to dismiss them as anything less than an essential information management 
tool, and they are the backbone of efficient, accurate, and useful data systems. 
Unfortunately, metadata are not accessible to data users unless the organization 
provides this access — generally through the development and implementation 
of a data dictionary or a more comprehensive metadata system. Such systems, 
when accompanied by a robust training program, serve as a critical link in an 
organization’s efforts to improve data use. Skills needed: Ensuring that data users 
have access to, and understand, metadata is an organizational responsibility — - from the 
development and implementation of an appropriate metadata system to relevant and robust 
staff training. Education professionals will need to learn how to access and navigate such a 
system, as well as understand any data or technical terms included in the metadata system 
(e.g., business rules, permitted values, and translation history). 


’ For more information about metadata, see the Forum Guide to Metadata: The Meaning Behind Education 
Data, available at http:/ / nces.ed.gov /forum/ pub_2009805 .asp. 
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What Do the Data Mean? 

The Graduation Rate in many states includes only those students receiving regular, 
standard, endorsed, or advanced diplomas. In contrast, the Completion Rate is generally 
used to measure graduates as well as students who receive a high school equivalency 
certificate, certificate of completion, or attendance certificate. The meaning of these 
similar data terms clearly depends on how the terms “graduate” and “completer” 
are defined, and anyone using the information would benefit from clear and accurate 
definitions for the terms. 


s View the data in an appropriate context. Because a single piece of data 
cannot possibly supply all the information needed to assess the status of, or 
change in, an education system, painting an accurate picture of the education 
enterprise requires a body of relevant data, with each individual data element 
imparting a complementary piece of the puzzle. For example, when evaluating 
trends in student performance data, it may be appropriate to consider data 
on student mobility rates, class size, teacher assignment, and instructional 
expenditure per pupil. When such a body of data is developed, the result is a 
well-integrated, multidimensional data set with an analytical value that may be 
much greater than the sum of its parts. Skills needed: Data users should become data 
literate in the broadest sense, developing an awareness of different types of data that might 
be relevant to the types oj questions that arise in their work. Good usage also requires 
developing analytical skills that permit the integration of data from across the education 
domain. Such integration is not necessarily quantitative in nature (literally combining data 
into new data sets), but can instead relate to the qualitative skill of assimilating various 
perspectives to inform decisionmaking and action. 


Why is context critical? 

A graduation rate of 40 percent might seem low, but what if it was at a school that 
specialized in trying to retain students who were at risk for dropping out? In such a 
case, context is critically important for interpreting a 40 percent graduation rate as 
tremendously successful in such a setting. 


S Recognize the nature of the collection tool. Not all data collection tools 
are created equal. In fact, skilled researchers sometimes produce data sets that 
are intentionally limited in their applicability because of choices made while 
designing the instrument. Depending on the scope of the project and purpose of 
the data, such a limitation can be entirely acceptable under certain circumstances. 
For example, a sample survey may be less expensive than a universe collection, 
but the findings may not be applicable to a broader population. Similarly, interim 
assessments can provide very important information about how well students 
have learned material in a given unit, but these assessments, by their nature, 
aren’t designed to be measures of content mastery for the purpose of grading 
at the course level. Skills needed: Understanding the nature of a collection tool can be 
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a complex undertaking. It is critical that data users have a thorough appreciation of the 
significant differences between major categories of collections (e.g., sample surveys versus 
universe collections) and assessments (e.g., interim versus summative assessments). While 
this knowledge is important to the fundamental tasks of being an education prfessional 
and should be learned in pre-service training, professional development on these topics is 
appropriate for many data users. 


s Identify bias in the collection source. Some organizations publish data 
about an issue for the purpose of advancing a specific position or philosophy. 

As such, data users need to evaluate the trustworthiness of their data sources. 
For example, might a report about the effectiveness of a particular instructional 
resource be viewed differently if the data originated in an objective source (e.g., 
another local education agency using an independent evaluator) as opposed to 
being published by the company that sells the product? Similarly, it is possible 
that a group with a policy agenda might present data that support a particular 
position rather than a more balanced and objective evaluation of an issue. Skills 
needed: Unlike some data skills that are heavily quantitative or analytical in nature, 
effectively evaluating the trustworthiness of a data source is mostly a function of effort: 
taking the time to research and apply common sense when evaluating the source. 


Do I really need to understand error? 

The effects of error measures can have substantial impacts on the actions of teachers. 

For example, depending on the uncertainty of the measures, students hovering just above 
and just below proficiency on state assessments may be equally at risk to score below 
proficient on future assessments. 


Timely data are useful 
data. 

Acting on ten-year- 
old data Is not likely 
to improve future 
educational outcomes, 
Make sure that the 
data you are using 
are timely enough to 
accurately describe 
current status and 
conditions. 


S Assess the quality of the data. Education professionals have access 
to a wide range of data to help inform action. Flowever, not all data are of 
comparable quality, and even when a data source is unbiased, it doesn’t mean 
that the data being presented are valid. Consider, for example, a research paper 
from a doctoral candidate who, in spite of good intentions, is still developing 
the expertise needed to collect, analyze, and present data that are accurate and 
reliable. Similarly, sometimes the results of a first time collection reflect the 
unfamiliarity of the organizations responding to a new form (e.g., new business 
rules, terms that may not be well defined, and elements that have not previously 
been collected or reported), which can have a negative effect on data quality. 
Skills needed: Assessing the validity and reliability of data may require considerable 
analytical training, but even a novice data user can improve the likelihood of sound usage 
by taking a few moments to consider some basic logical requirements of good data quality A 
• Is the source of the data valid (i.e., is it a good tool for measuring the issue at hand)? 

For example, an end-of-year science assessment that only tests content from the last 


3 For more information about data quality, see the Forum Guide Co Education Indicators, available at 
http: / / nces.ed.gov/forum/pub_2005 802 .asp. 
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eight weeks of the academic calendar is not likely to be a valid measure of student 
learning for the year. 

Is the source of the data reliable (consistent)? For example, if a survey were to be 
administered to similar groups on dfferent days, how likely is it that respondents 
would provide the same responses ? If it is not likely to generate similar findings, the 
instrument may not be reliable and the quality of the data is uncertain. 

What utility do the data have? If the data have been collected, but never used by the 
stakeholders, then data quality may be questionable. 


V Appreciate the assumptions and limitations of the data. Data may 

be collected in a manner tint constrains their applicability for use. Oftentimes, 
information about data meaning, format, and usage are available as metadata. 

For example, a “cohort graduation rate” is intended for assessing student success 
with respect to meeting the expectations for a regular diploma, but it is not 
a good measure of the number of students w ho completed high school in a 
particular year because it excludes students who did not graduate on time and 
students w ho earn certificates of attendance or high school equivalency degrees. 
Similarly, the results of a norm-referenced test are usually not a good instrument 
for evaluating changes to the instructional program precisely because norm- 
referenced instruments are intentionally designed to be relatively stable around 
a normal curve and less sensitive to variation in instructional practices. Good 
data use requires an understanding of the assumptions and limitations of each 
piece of data. Skills needed: Inquire about information related to the collection, source, 
and definitions of the data. This may be available online or as part of a report, but may also 
require discussion with data managers and data stewards who can verify that the data are 
being used appropriately . + 


Trends (multiple points 
of data pointing toward 
the same general 
trend/conclusion) are 
more powerful than 
single data points that 
aren't supported by 
other data. 


V Comprehend the significance of error and/or uncertainty in data. 

The statistical nature of some data collections (sample surveys, assessment 
results, etc.) means that some measurements reflect a certain amount of 
uncertainty. Confidence intervals and other methods of describing this 
uncertainty provide important information about the limits of data accuracy. 
While all data users do not need to understand the statistical methods for 
calculating error, they should understand that measurements often have limits 
to their accuracy. Skills needed:While calculating error requires considerable statistical 
training, even a novice data user can improve the likelihood of sound usage by taking a few 
moments to consider the implications of error measures. 


4 Forum Guide to Building a Culture of Quality Data: A School and District Resource , available at 
http: / / nces.ed.gov/Jorum/puh_200S 801 .asp. 


Overview 


11 



...a wi 








Figure 3. Because of the widespread 
use of tabular and graphical data, data 
users should become comfortable with, 
and even skilled at, analyzing data in 
tables and figures. 


S Discern meaning in tabular and graphical data. Tabular data 

refers to the presentation of data in rows and columns, a practice that is 
common in education data reporting as well as many other fields. Graphs and 
figures are another frequently used method of presenting data. While graphs 
can, theoretically, occur in multiple dimensions, most reporting of education 
data focuses on two dimensions, usually referred to as the x- and y-axes in 
graphs. Other common graphical presentations include bar graphs, pie charts, 
and area charts. Skills needed: Because of the widespread use of tabular and graphical 
data, data users should become comfortable with, and even skilled at, analyzing data in 
tables and figures. Interpreting tabular data requires that a user pay close attention to 
column and row titles as well as footnotes and other appended materials that often are 
included in tables to help clarfy the presentation and meaning of the data. Reading a 
graph can be fairly straightforward, but understanding the relationship between the x- 
andy-axis data points requires some background knowledge. For example, the commonly 
cited statistical adage “correlation is not causation” is particularly applicable to 
graphical data — -just because there appears to be a visual relationship between points 
on a graph does not mean that one variable caused the other to occur. As with tabular 
data, graphical data appear frequently in the field of education (e.g., an outcome 
measure reported over time). Similarly, pie charts and area charts are frequently used as 
a way of presenting data in a format that is more intuitively understood by users than 
tables of raw data. 

S Know how to effectively question data/report authors. Many 

data users do not feel they have the authority to question the “experts” who 
produce research reports and data sets. After all, these researchers often have 
advanced degrees in research and evaluation, statistics, education policy, or 
other fields of specialization. Nonetheless, when a data report or research 
finding is not clearly presented, at least with respect to the practitioners’ 
needs, it is entirely appropriate for a data user to contact the source of the 
data to request clarification or correction. Skills needed: Unlike some data skills 
that are heavily quantitative or analytical in nature, effectively questioning a data 
source is mostly a function of effort and confdence. 
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A Question-Driven Approach to Data Use 

This document recommends a question-driven approach to data use. For any given scenario 
that requires a stakeholder to use data to take action, the following questions can help to 
evaluate data needs and analysis — and inform decisionmaking and action. Table 1 presents 
an application of these questions in an example scenario. 

V What do I want to know? 

V What data might be relevant? 

V How will I access the relevant data? 

V What skills and tools do I need to analyze the data? 

V What do the data tell me? 

V What are my conclusions? 

V What will I do? 

A What effects did my actions have? 

K What are the next steps? 


In a 2007 survey, 
less than two-thirds of 
teachers reported using 
data from a student 
data system even a 
few times a year to 
track individual student 
test scores, estimate 
whether students were 
making adequate 
progress, identify gaps 
in skill development, 
or inform student 
placement in courses 
or special programs 5 


Data Use Recommendations for Key Audiences 

Education data are growing in quantity, quality, and value. When appropriately used to 
guide action, data can be a powerful tool for improving school operations, teaching, 
and learning. Education stakeholders who possess the knowledge, skills, and abilities to 
appropriately access, analyze, and interpret data will be able to use data to take action 
that benefits students, schools, and the entire education enterprise. See appendix A for 
examples of online resources that are readily available to enterprising individuals who 
understand the value of strong data use skills. 

Whether data users have access to sophisticated data analysis solutions or simple reporting 
tools, this document is intended to provide helpful guidance for using data to take action. 
The briefs that accompany this Introduction are written for three key education audiences: 
Educators, School and District Leaders, and State Program Staff. Other likely audiences of 
this document include early childhood education leaders, higher education leaders, social 
services agencies, federal education staff, education policymakers, researchers, parents, and 
other education stakeholders. 


5 Source: U.S. Department of Education, available at 

http: / / www.ed.gov/ about/ offices / list / opepd/ ppss/ reports, htmli^edtech 
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Table 1. An illustration of how the question-driven approach to data use can be applied by teachers. 6 


" £ 

£ >> 
o U 

Q 


.. £ 
*— c 
a v 
£ 

co <D 
co 


D 

O 


a o 
a> o 


<1> 

u 

u 

< 


o 

Q 

Q) 

* B- 

Q. <1> 

£ £ 
co \ 
<D 

N 

>. 

o 

C 

< 


o 

o 

.. Q 

<D 

a £ 

£ c 
</> o 

u 

< 


.. a> 
10 o 
a s 
o 

co > 


Framework 

Scenario 

Comments 

Scenario 

A teacher observes that a student is not adding 
fractions properly. 

An example of an academic situation that is likely to 
occasionally arise in a classroom setting. 

What do 1 want 
to know? 

Based on this scenario, 1 need to know... 

Why is the student adding fractions incorrectly 
and what needs to be done to improve this skill? 

Additional/secondary questions may also be relevant. 

What data 
might be 
relevant? 

Based on the information 1 need, possibilities 
include... 

0 observational and anecdotal data 
□ behavioral/discipline data 

0 homework and classwork records 

Although there may be a wealth of data that could 
be evaluated, the scenario may warrant only a small 
subset of data for analysis. 

The selected boxes indicate the most relevant options 
to address this example/scenario. 

How will 1 
access the 
relevant data? 

Based on the relevant data, 1 will... 

0 consult the student's portfolio 
□ interview the student's parents about 
observational and anecdotal data 

You may already have access to these data sources; 
other needs may require gathering data you do not 
already have. The process for accessing these data 
may vary by organization. The selected boxes indicate 
the most relevant options to address this example/ 
scenario. 

What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will 
need to be able to... 

0 relate, combine, and balance data 
from different sources 

0 assess the data quality and limitations 

Don't be too quick to form an opinion based on one 
or even a few pieces of data. Use techniques such as 
triangulation to assimilate and evaluate the body of 
data in order to develop conclusions. 

What do the 
data tell me? 

Based on this analysis, the data suggest... 

□ inconsistent or conflicting results 

□ an isolated event 

0 a trend 

Is there a correlational relationship in the data? Is there 
a cause-effect relationship in the data? If it is a single 
incident, the teacher may choose to note it but do 
nothing more at this time. The selected boxes indicate 
the most relevant options to address this example/ 
scenario. 

What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

The student missed several lessons when the 
concept was originally being taught and the 
student shows only partial understanding. 

If it is not a problem that requires action, then it is the 
end of the story. 

If it is a problem that requires action, then more 
information becomes necessary. 

What will 1 do? 

Based on this conclusion, actions 1 will take 
include... 

□ find more information from other sources 
0 teach the material again or differently 

What support is currently available to the student? 

Are there interventions 1 can propose to help the 
student? 

The selected boxes indicate the most relevant options 
to address this example/scenario. 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 

□ the student does not demonstrate 
improvement 

0 the student demonstrates that he or she 
now adds fractions at an appropriate 
skill level 

At a later point in time, 1 can assess whether my action 
(based on data use) changed the situation. 

The selected boxes indicate the most relevant options 
to address this example/scenario. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

0 observe whether the student continues 
to demonstrate adequate mastery of 
fractions 

If the situation is not completely resolved, the educator 
may wish to identify what specific issues still need to be 
addressed and return to the beginning of the data use 
cycle to seek new information that will inform action. 


6 In some cases, you may collapse these steps in your day-to-day application of this framework. For example, if you have taught a student for 
several months, you might already know the student’s profile. In other instances, you may wish to rely on this framework more precisely as you 
formally analyze the data. 
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Brief I: Data Use for Educators 


Educators use data to better address student needs and 
classroom management 

As an educator, you collect and use data all the time. Often, these data are real-time and 
observational in nature. For example, you might ask students to raise their hands if they 
think they know the meaning of a vocabulary word, or you might look over their shoulders 
at answers they are calculating on a math problem — and you are using these observational 
data to determine your next actions in the classroom. Particularly good teachers with a 
lot of experience may even be able to assess whether students understand the material 
just by watching the class’s response to certain instructional cues. Just as these examples 
of “informal” data use help to improve teaching and learning in the classroom, other data 
can be used to actively inform instructional planning, classroom management, curriculum 
choices, and communication with school and district administrators. 

Educators access and analyze data for many reasons. For example, with the right data, you can 

"S identify areas of the curriculum in which a class or individual students are 
succeeding or struggling; 

modify instruction for individuals or groups of students based on their learning 
needs, and adjust or adapt lesson plans or assignments as needed to differentiate 
classroom-level needs (e.g. , revisit content areas, slow or speed up the pace of 
the curriculum, or enrich coursework); 

^ set or adjust classroom-level improvement goals; 

"S identify practices that have shown the most positive effect on student learning; 

"S identify students demonstrating signs of engagement or disengagement and how 
it affects their behavior and academic performance; 

"S collaborate with (and learn from) other educators when data suggest they are 
particularly effective in certain subjects or with specific instructional 
practices; and 

"S communicate effectively with school administrators, district leaders, and peers in 
other classrooms. 


With the recent 
development of more 
robust data systems, 
ranging from classroom 
data dashboards to 
statewide longitudinal 
data systems, 
educators have access 
to more data than ever 
before. The challenge 
is to use these data to 
improve teaching and 
learning rather than 
being overwhelmed by 
the possibilities of 
data use. 
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Data use in the classroom is not new. Historically, educators have been asked to use data 
to answer a wide range of questions: 

'k Instruction : e.g., Do these spelling (or other subject matter area) test results 
suggest that a student is ready to proceed to the next lesson, or should I provide 
some review exercises? 

S Classroom Management : e.g., How many milk orders do I need for my 
kindergarten class today? 

'k Compliance : e.g., Am I giving enough time to vocabulary lessons to meet the 
district’s stated policy on minutes on task? 

'k Accountability: e.g., Is my class achieving our performance goals? 


Steps to seeking 

information include 

'k Recognizing when 
information is 
needed to inform a 
decision and direct 
an action. 

'k Defining a 

meaningful and 
achievable scope 
of concern/unit of 
analysis. 

'k Clearly articulating 
the issue at 
hand as a critical 
question. 

'k Assessing what 
relevant data are 
already available. 

'k Identifying "gaps" in 
currently available 
data. 

’k Identifying potential 
barriers to finding 
the information and 
ways to overcome 
them. 

'k Seeking additional 
data that are 
accurate, valid, 
reliable, timely, and 
relevant. 


The Cycle of Data Use 

How do people use data? 


Wise data use should help to increase the effectiveness of an educator’s primary purpose: 
instructing students. The cycle of data use presented in figure 1 . 1 has data use to inform 
action as its primary goal. As such, the cycle is based on a framework consisting of five 
iterative phases (or activities): (1) seek information; (2) access/ gather data; (3) analyze/ 
interpret data; (4) act; and (5) evaluate. Such a framework can serve as the basis for the 
systematic use of education data in classrooms and schools. 



2. Access/ 
Gather Data 


1. Seek 
Information 


3. Analyze/ 
Interpret Data 


Evaluate 


4. Act 


Figure 1.1 The Cycle of Data Use. For a deeper discussion about the cycle of data, see the 
Introduction to the series Forum Guide to Taking Action with Education Data, available at 
http://nces.ed.gov/forum/publications. 
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1. Seek Information 

“What data do I have versus what data do I need?” 

“Wise data use” requires that educators be efficient in their search for useful data. To be 
effective, data use should be driven by well-defined questions. For example, rather than 
trying to find data to address a very broad question such as, “Flow can I improve reading?” 
it is likely that a more precisely worded question, such as “Are students demonstrating 
that they are comprehending reading tasks?” will enable you to find more relevant data to 
inform planning and action. 


When faced with such a question, ask yourself, “What information do I need (and not 
currently have) to answer this question?” Identifying these gaps will point you toward 
different types of data depending on the information gap you hope to address. For example, 
with the question above about reading comprehension, in addition to formative assessment 
results (data you already have), you might seek data related to language and language arts, 
such as vocabulary scores, language fluency, language proficiency, and reading data from 
previous years of schooling. 

2. Access/Gather Data 

“What data do I have and what data can I get?” 

As an educator, what data do you use from national, state, regional, district, or school 
data systems? Are these data accessible via downloadable data files, grade books, reports, 
spreadsheets, a data warehouse, a student management system’s portals, dashboards, 
analysis instruments, or graphical visualization tools? In addition to information collected 
about students (e.g. , observations, informal assessments, homework assignments, etc.), you 
may be able to access a host of data from annual summative assessments and more frequent 
interim and formative assessments, as well as demographic, attendance, discipline, program 
participation, and other datasets. Depending on your information needs, these may be 
aggregate level (e.g., grade-level or building-level ) or individual-level (e.g., Student First 
Name), snapshot or longitudinal in nature, observational or recorded, and may be in the 
form of raw numbers for analysis or customized reports that present data specifically for 
the easy use of educators. 

For example, suppose a parent challenges the final grade given to a student in a course you 
teach. As the educator responsible for the grade, you might be expected to explain your 
decision. In this scenario, it is likely that the data access/ gathering phase might focus on 
a host of data from your electronic grade book, including exam scores, quiz scores, and 
homework points. Flowever, many teachers also include other sources of information when 
calculating grades, including class attendance (from the student management system), 
class participation (from handwritten notes), projects and presentations (from a portfolio), 
and academic improvement over time (from the student record system). Collecting these 
data often requires engaging numerous data sources, and sometimes the challenge is not 


Steps to accessing/ 
gathering data include 

S identifying whether 
the needed data 
already exist. 

S Collecting the data, 
which often means 
gaining access 
to relevant data 
that have already 
been collected by 
someone else. 

S Learning about 
the data, such 
as applicable 
definitions, 
limitations, 
timeliness, 
formatting, other 
data sets, etc. 
(often referred to 
as metadata), to 
ensure that you 
fully understand 
the information. 
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in finding the data, but understanding them (see Step 3: Analyze /Interpret Data). Gaining 
access to data may also be a challenge if you request personally identifiable data. You should 
always consider whether de-identified or aggregate data would meet your needs. 

3. Analyze/Interpret Data 

“What do the data suggest?” 


Steps to analyzing 
and interpreting data 
include 

S Formatting the 
data in a manner 
that enables sound 
analysis and 
interpretation. 

S Determining any 
limitations on, or 
constraints to, 
data use, including 
consideration of 
the data's 

■ purpose; 

■ quality; 

■ bias; and 

■ timeliness. 

S Reviewing whether 
conclusions are 
logically and 
statistically sound 
and defensible. 


Having access to needed, relevant data is necessary, but not sufficient, for appropriate 
data use. You must also be able to skillfully analyze and take appropriate action based upon 
the information to change instructional or classroom management practices. Research 
has shown these data skills to be the areas of training that teachers are the least likely 
to receive,’ but strengthening these skills is possible through individual effort as well as 
thoughtful professional development targeted to educators by local and state education 
agencies. Educators who are good data users usually understand 

A the meaning of the data (e.g., interim assessment results are not the same as 
summative assessment results); 

■A commonly used measures for school and student performance reporting (e.g., a 
scale score is not the same as a percentile score); 

^ basic statistical concepts (e.g., mean, median, and mode); 

■A trends (e.g., multiple points of data pointing toward the same general trend or 
conclusion are more powerful than one or two data points that aren’t supported 
by other data) ; 

■A common presentation formats (e.g. , tabular reports as well as graphs and charts); 

•A assumptions that limit the appropriate use of data (e.g., a good reading score 
does not automatically mean that the student is strong in all language arts 
areas); and 

A the value of collaborating with colleagues (e.g., instructional staff relying on data 
staff to advance understanding) . 

Failure to understand these aspects of data can result in either ignoring data completely 
when making decisions or, perhaps worse, using data confidently but incorrectly (e.g., 
misinterpretation, over-interpretation, or inappropriate application). 


7 On the NETTS teacher survey (U.S. Department of Education 2009), the majority of teachers with access to 
student data systems reported that they use the systems to provide information to parents (68 percent), track 
individual test scores, and monitor student progress (both 65 percent). Less common uses of data — reported 
by just over a third of the teacher survey respondents — were identifying promising practices (39 percent), 
informing student placement in courses or special programs (37 percent), and assessing test taking needs (36 
percent). Source: LI.S. Department of Education, Office of Planning, Evaluation, and Policy Development, 
Use of Education Data at the Local Level From Accountability to Instructional Improvement, Washington, 
D.C., 2010, available at http:/ / www2.ed.gov/rschstat/eval/tech/use-oJ-education-data/use-oJ-education-data.pdJ. 
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Figure 1.2 Analysis includes looking for patterns in the data. Rather than finding one piece 
of data and using it to make a generalization or draw other conclusions that are beyond the 
scope of the single data point, identify multiple data sources that inform the analysis of the 
questions— a process sometimes referred to as “triangulation.” 


4. Act 

“How can data be acted upon to improve educational processes and outcomes?” 

By definition, “actionable” data refer to information that allows a decision to be made and 
action to be taken. When you, as an educator, are data literate, you are able to supplement 
your education expertise and personal experience in the classroom with the right data for 
the right purposes: to improve instruction and performance. For example, assessments in 
many states are used to determine whether academic performance targets are being met 
for accountability purposes. Flowever, those same assessments often generate item response 
data for individual students or groups of students. When assessments are designed to 
provide meaningful feedback at this level, teachers, instructional support staff, and building 
principals can work with data staff to identify concepts that students have mastered, as well 
as those topics that may warrant review. This analysis can occur for an individual student, 
groups of students (e.g. , by common subgroups such as race/ ethnicity, language learners, 
and socioeconomic status), and at the classroom and grade levels. Trends in these data can 
then be used to assess the relative strengths and weaknesses of the curriculum and various 
instructional strategies. Based on these findings, you may even change or supplement the 
curriculum as necessary to better meet the specific learning needs of your students and 
classes. 
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5. Evaluate 

“Has the issue changed for the better?” 

Actionable data are powerful tools that often lead to change. The question is whether 
this change is for the better and whether the situation has improved sufficiently to have 
adequately addressed the original concern. For example, if you apply the data use cycle to 
help a student improve reading comprehension, it is critical to learn whether the student 
did, in fact, improve. But it is equally important to assess whether that improvement 
resulted in the student achieving expected growth targets rather than just building 
rudimentary skills that are still not satisfactory. As such, determining whether the original 
issue has been adequately addressed is an important component of the data use process. This 
type of evaluation often includes consideration of the following types of questions: 

S Flow effectively has the initial issue been resolved? 

'E What new concerns have arisen? 

V Which factors are understood sufficiently and which warrant 
additional investigation? 

V Flave new data gaps (needs) been identified? 

If the issue has been effectively addressed by data-informed action, then you may wish to 
share your experience with colleagues and encourage them to use data to deal with similar 
problems. If the issue hasn’t been fully resolved, the cycle of data use repeats (see Step 1 : 
Seek Information). 

Critical Data Use Skills 

As an educator, you were trained to teach students and assess academic progress. Data 
are becoming increasingly relevant to these duties. Some fundamental knowledge, skills, 
and abilities needed to effectively analyze and interpret data — and thus use education data 
effectively — include 

V Understand what the data mean. Is a completion rate the same as a 
graduation rate? Do completion rate data in a table refer to a point in time 
during a particular academic year, or an average over several years? Data users 
have difficulty knowing what the data really mean unless they understand data 
definitions, how the data are collected and formatted, and other usage guidance 
and limitations. Data about data are often referred to as “metadata.” 8 

V View data in an appropriate context. Data users should avoid jumping to 
a conclusion based on a single piece of data without being sure they understand 
the context of the data. When you evaluate data, think about complementary 
information that paints a richer picture of a student’s education experience. For 


8 Forum Guide to Metadata: The Meaning Behind Education Data , available at 
http: / / nces.ed.gov /Jorum / pub_2009805 .asp . 
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example, if you notice that a student misbehaves every day at 1 1 :00 a.m. during 
language arts class, it is possible that the student doesn’t like language arts, but 
given that the behavior occurs at the same time each day, it is also possible that 
something other than language arts is the problem. Could it relate to external 
factors such as the student not eating breakfast in the morning or anxiety prior to 
the lunch hour where, perhaps, the student is being bullied? The point is that the 
inappropriate behavior occurs within the broader context of the student’s life, 
and remedying the situation may require data that reflect that broader context. 


What Do the Data Mean? 

The Graduation Rate in many states includes only those students receiving regular, 
standard, endorsed, or advanced diplomas. In contrast, the Completion Rate is generally 
used to measure graduates as well as students who receive a high school equivalency 
certificate, certificate of completion, or attendance certificate. The meaning of these 
similar data terms clearly depends on how the terms “graduate” and “completer” 
are defined, and anyone using the information would benefit from clear and accurate 
definitions for the terms. 


s Recognize the nature of the collection tool. Most data collection tools are 
designed to generate data for a specific purpose. For example, some assessments 
measure skill development across a broad spectrum, others evaluate content 
mastery relative to a standard criteria, and still others are used to rank student 
performance relative to other students. Similarly, some reading assessments 
are especially sensitive to skill development in emerging readers but less 
effective indicators of progress being made by strong readers. Thus, each type of 
assessment is constructed for a different purpose, which should inform how the 
results are used. 


Why Does Context Matter? 

Temperature is a measurement most people think they understand, but to correctly 
interpret it as information requires additional context. 

• What is the scale? Degrees Celsius or Fahrenheit? 

• Is it the high or low temperature for the day? 

• Is it the current temperature, yesterday’s temperature, or tomorrow’s 
predicted temperature? 

This context is critical to using the piece of data as information. In its absence, “28” could 
mean 28° Fahrenheit as the day’s high temperature (brrr, is it cold) or 28° Celsius (the 
equivalent of 82° Fahrenheit) as tomorrow’s predicted low temperature (wow, will it ever 
be a scorcher). 
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s Identify bias in the collection source. Be wary of data sources that may 
have a bias. For example, what if you were given data that showed that buying 
a piece of instructional software could substantially raise your district’s reading 
scores? As an educator, would you view the data differently if the data originated 
in an objective source (e.g. , a colleague in the neighboring district) as opposed to 
being published by the company that sells the product? 


Do I really need to understand error? 

The effects of error measures can have substantial impacts on the actions of teachers. 

For example, depending on the uncertainty of the measures, students hovering just above 
and just below proficiency on state assessments may be equally at risk to score below 
proficient on future assessments. 


Timely data are useful 
data. Acting on ten- 
year-old data Is not 
likely to Improve future 
educational outcomes. 
Make sure that the 
data you are using 
are timely enough to 
describe current status 
and conditions. 


"S ASSGSS the quality of the data. Data quality will limit the effectiveness 
of actions based on data. In other words, are your data accurate, reliable, and 
timely? For example, an end-of-year science assessment that only tests content 
from the last eight weeks of the academic calendar is not likely to be a valid 
measure of student learning for the entire year. Similarly, if a survey were to be 
administered to similar groups on different days, how likely is it that respondents 
would reliably provide the same responses? If it is not likely to generate similar 
findings, the instrument may not be reliable and the quality of the data is 
uncertain . 9 

S Appreciate the assumptions and limitations of data. Oftentimes, 

information about data meaning, format, and usage are available as metadata 
(data about data) . For example, the results of a norm-referenced test are 
designed to reflect a normal curve, meaning that such an assessment is usually 
not a good tool for evaluating changes to the instructional program. Good data 
use requires an understanding of the assumptions and limitations of each piece of 
data. As an educator who needs to act on data, you should expect to collaborate 
with data managers, data stewards, or other staff responsible for explaining the 
intricacies of data elements, formulae, definitions, formats, and meaning. 

S Comprehend the significance of error and/or uncertainty in data. 

While calculating error requires considerable statistical training, even a 
novice data user can improve the likelihood of sound usage by taking a few 
moments to consider the implications of error measures. As an educator, you 
should understand that confidence intervals and other methods of describing 
uncertainty provide important information about the limits of data accuracy. 


9 Forum Curriculum for Improving Education Data: A Resource for Local Education Agencies, available at 
http: / / nces. ed.gov/Jorum / pub_2007808 .asp. 
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K Discern meaning in tabular and graphical data. Because of 

the widespread use of tabular and graphical data in school, district, 
and state reporting, you will want to become comfortable with, 
and skilled at, analyzing data in these formats. Interpreting tabular 
data requires that you pay close attention to column and row titles, 
as well as footnotes and other appended materials, which often 
provide the context in which to interpret meaning. Reading a graph 
can be fairly straightforward, but understanding the relationship 
between the x- and y-axis requires some training. For example, the 
commonly cited statistical maxim, “correlation is not causation” is 
particularly applicable to graphical data — just because there is a 
visual relationship between points on a graph does not mean that 
one variable caused the other to occur. 

K Know how to effectively question data/report authors. 

Sometimes the class roster you receive from the district is wrong. 
Other times, research reports draw conclusions that don’t seem 
obvious from the data. Flow does an educator deal with inaccurate, 
misleading, or poorly presented data? Unlike some data skills 
that are heavily quantitative or analytical in nature, effectively 
questioning a data source is mostly a function of effort and 
confidence. When a data report or research finding appears to be 
inaccurate, misleading, or unclear to you, a practicing educator, 
then it is entirely appropriate to contact the source of the data to 
politely and professionally request clarification or correction. In 
addition to addressing your immediate concerns, you may help the 
author make the data more accurate, understandable, and actionable 
for the entire education community. 



Figure 1.3. Because of the widespread 
use of tabular and graphical data, data 
users should become comfortable with, 
and even skilled at, analyzing data in 
tables and figures. 


Training and Skill Development 

Ideally, educator preparation programs would begin the process of developing data use 
skills. Current professionals might receive informal support through peer-to-peer study and 
professional learning communities. A great deal of information concerning the processes 
and techniques that support good data use can also be found online by enterprising 
individuals who understand the value of strong data use skills. More formally, professional 
development opportunities might include training that emphasize topics such as recognizing 
whether data are within acceptable ranges, identifying patterns or trends in the data, 
and applying this information to solve real-world problems. Similarly, trainers with data 
expertise can develop a staff’s quantitative skills, such as appreciating the meaning of error 
measures; calculating the amount of change in a variable presented in charts, tables, and 
graphs; and interpreting the significance of statistical summaries of large data. 
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A Question-Driven Approach to Data Use 

Using the skills described in the previous section, educators can rely on a question-driven 
approach to data use for nearly any scenario that requires making a data-informed decision 
and taking data-informed action. 

A What do I want to know? 

A What data might be relevant? 

A How will I access the relevant data? 

A What skills and tools do I need to analyze the data? 

A What do the data tell me? 

A What are my conclusions? 

A What will I do? 

A What effects did my actions have? 

A What are the next steps? 

Tables 1 and 2 illustrate how these questions can be applied to example scenarios 
commonly seen by educators. 
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Table 1. An illustration of a question-driven approach to data use for various scenarios facing educators (example l) 10 



Framework 

Scenario 

Comments 

2 <D 
a * 

Scenario 

A teacher observes that a student's math 

An example of an academic situation that is likely to 



performance is declining. 

occasionally arise in a classroom setting. 


What do 

Based on this scenario, 1 need to know... 

Additional/secondary questions may also be 

£ 

1 want to 



relevant. 


know? 

Is the declining math performance part of a broader 


jU <D 


pattern of declining performance or related to a 


co 


specific issue? 



What data 

Based on the information 1 need, possibilities 

Although there may be a wealth of data that could 


might be 

include. 


be evaluated, the scenario may warrant only a small 


relevant? 

0 

observational and anecdotal data 

subset of data for analysis. 



□ 

health information 




□ 

family information 




□ 

behavioral/discipline data 




S 

homework and classwork records 




S 

interim benchmarks 

11 past math test results 

0 classroom assessments (projects, quizzes) 


O 


S 

historical end-of-year math tests 


o 


□ 

other assessment data 


.. a> 


□ 

student attendance records 


CN £ 

a o 


□ 

teacher assignment/attendance data 








How will 1 

Based on the relevant data, 1 will... 

You may already have access to these data 

s 

access the 



sources; other needs may require gathering data 

u 

relevant 


consult colleagues about observational 

you do not already have. 

< 

data? 


and anecdotal data 

reflect on my recent experiences with the 

The process for accessing these data may vary by 




student 

organization. 




consult the student's portfolio to review 
past work 





query the student information system for 
interim benchmark data 





use my gradebook to review the student's 
homework data 




□ 

ask the student's parents about 
observational/anecdotal data 



What skills 

Based on gaining access to these data, 1 will need to 

Don't be too quick to form an opinion based on one 


and tools 

be able to... 

or even a few pieces of data. Use techniques such 


do 1 need to 



as triangulation to assimilate and evaluate the body 


analyze the 

S 

understand the context and how the data 

of data in order to develop conclusions. 

o 

data? 


are presented 


a 


S 

organize the data in a way that applies to 


a5 



this situation 


Step 3: 
e/lnterpr 


S 

relate, combine, and balance information 
from different sources 



S 

assess the data quality and limitations 


>* 

What do the 

Based on this analysis, the data suggest... 

Is there a correlational relationship in the data? 

C 

data tell me? 




< 


□ 

inconsistent or conflicting results 

Is there a cause-effect relationship in the data? 



□ 

an outlier 




□ 

an isolated event 




S 

a trend 



10 In some cases, you may collapse these steps in your day-to-day application of this framework. For example, if you have taught a student for 
several months, you might already know the student’s profile. In other instances, you may wish to rely on this framework more precisely as you 
formally analyze the data. 
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Step 5: Step 4: 

Evaluate Act on the Data 



Table 1. An illustration of a question-driven approach to data use for various scenarios facing educators (continued) 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

The student missed a fundamental concept that was 
taught at the beginning of the year and is showing 
only partial understanding. 

If it is not a problem that requires action, then it is 
the end of the inquiry. If it is a problem that requires 
action, then more information becomes necessary. 

What will 1 
do? 

Based on this conclusion, actions 1 will take include... 

What support is currently available to the student? 


□ find more information (check other data 

Are there interventions 1 can propose to help the 


sources) 

□ teach the material again (doing again 
rather than differently) 

HI spend extra time with the student 

HI send extra practice exercises home 

□ recommending changes to the curriculum 

HI communicate with: 

HI the student 

□ the student's parents 

HI instructional support staff 

□ curriculum advisors 

student? 

What effects 

Based on these actions, 1 observed that... 

At a later point in time, 1 can assess whether my 

did my 


action (based on data use) changed the situation. 

actions have? 

□ the student did not demonstrate 
improvement 

□ the student demonstrated partial 
improvement 

HI the student demonstrated that he or she 
now understands the concept at an 
appropriate skill level 


What are the 

Based on my evaluation, 1 will... 

If the situation is not completely resolved, the 

next steps? 


educator may wish to identify what specific issues still 


HI continue to monitor the situation 

need to be addressed and return to the beginning 


HI observe whether the student continues to 

of the data use cycle to seek new information that 


demonstrate adequate mastery of fractions 
HI determine whether additional intervention 
is needed 

HI observe whether other students are having 
this problem 

HI if other students are not having this 
problem, then the response (action) will 
likely focus on the individual student 

HI if other students are having this problem, 
then the response (action) may be an 
instructional or curriculum issue 

HI evaluate whether subsequent questions 
have arisen and, if necessary, use this 
question-driven approach to data use to 
advance the inquiry 

will inform future action. 
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Table 2. An illustration of a question-driven approach to data use for various scenarios facing educators (example 2) 11 



Framework 

Scenario 

Comments 

Data Use 
Cycle 

Scenario 

A 5th grade teacher is preparing an instructional 
plan for the upcoming school year and wants to 
know whether last year's plan was effective. 

A common occurrence for educators. Good 
teachers often customize previously successful 
instructional plans to meet the learning needs of 
incoming students. 

Step 1: 

Seek Info 

What do 

1 want to 
know? 

Based on this scenario, 1 need to know... 

Did the students in my classroom last year 
demonstrate sufficient academic progress? 

Subsequent questions include: What is the 
academic and behavioral profile of individual 
students in the class? 

Are there particular academic and behavioral 
situations to accommodate? 

What modifications should 1 make to last year's 
instructional plan? 

Step 2: 

ess/Gather Data 

What data 
might be 
relevant? 

Based on the information 1 need, possibilities 
include... 

S Academic performance data 

11 4th grade summative assessments 

11 5th grade summative assessments 

11 Academic growth data 

11 Baseline performance data 

11 Past trajectory performance data 

□ Previous classroom experience data 
□ Anecdotal data concerning the 

strengths weaknesses of teachers 

11 Program participation data 

11 Students identified as having special 
learning needs 

11 Standards data 

□ Mobility data 

□ Attendance data 

□ Discipline data 

11 Curriculum data 

11 Curriculum changes 

Although there may be a wealth of data that could 
be evaluated, the scenario may warrant only a small 
subset of data for analysis. 

u 

o 

< 

How will 1 
access the 
relevant 
data? 

Based on the relevant data, 1 will... 

□ consult colleagues about observational 
and anecdotal data 

11 reflect on my recent experiences with the 
students 

11 consult the students' portfolio to review 
past work 

□ query the student information system for 
interim benchmark data 

11 query the student information system for 
summative data 

11 query the student information system for 
program participation data 

□ use my gradebook to review the students' 
homework data 

You may already have access to these data 
sources: other needs may require gathering data 
you do not already have. 

The process for accessing these data may vary by 
organization. 


11 In some cases, you may collapse these steps in your day-to-day application of this framework. For example, if you have taught a student for 
several months, you might already know the student’s profile. In other instances, you may wish to rely on this framework more precisely as you 
formally analyze the data. 
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Step 5: Step 4: Step 3: 

Evaluate Act on the Data Analyze/Interpret Data 



Table 2. An illustration of a question-driven approach to data use for various scenarios facing educators (continued) 


What skills 
and tools 
do 1 need to 
analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 

E understand the context and how the data 
are presented 

E organize the data in a way that applies to 
this situation 

E relate, combine, and balance information 
from different sources 

E assess the data quality and limitations 

Don't be too quick to form an opinion based on one 
or even a few pieces of data. Use techniques such 
as triangulation to assimilate and evaluate the body 
of data in order to develop conclusions. 

What do the 
data tell me? 

Based on this analysis, the data suggest... 

E the students in my classroom last year 

demonstrated sufficient academic progress 

□ the students in my classroom last year did 
not demonstrate sufficient academic 
progress 

E there are curriculum changes that demand 
changes to the instructional plan 

□ there are not curriculum changes that 
demand changes to the instructional plan 

Are there gaps in the data? In other words, are 
data that would be helpful to evaluate missing or 
otherwise unavailable? If so, who needs to be told 
this information so that it can be corrected in the 
future? 

What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

□ the instructional plan from last year was not 
effective 

E the instructional plan from last year was 
effective 

E curriculum changes need to be integrated 
into the instructional plan 

If it is not an issue that requires action, then it is the 
end of the story, if it is an issue that requires action, 
then more information becomes necessary. 

What will 1 
do? 

Based on this conclusion, actions 1 will take include... 

□ do not modify last year's instructional plan 

E make focused /specific changes to last 

year's instructional plan 

□ create a new instructional plan 

□ identify deficiencies in the curriculum 

□ find more information (check other data 
sources) 

Although the instructional plan may appear to 
be successful, it is possible that changes to the 
curriculum could warrant updates to the plan. 

What effects 
did my 
actions 
have? 

Based on these actions, 1 observed that... 

E the revised instructional plan is appropriate 
for the new class 

□ the revised instructional plan is not 
appropriate for the new class 

At a later point in time, 1 can assess whether my 
action (based on data use) was successful. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

E observe whether students in the class 
continue to demonstrate sufficient 
academic progress 

E assess the specific needs of future incoming 
classes 

If the situation is not completely resolved, the 
educator may wish to identify what specific issues still 
need to be addressed and return to the beginning 
of the data use cycle to seek new information that 
will inform future action. 
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Conclusions 

High quality and timely data are powerful tools for improving teaching, learning, and the 
entire education enterprise. Thoughtful application of the frameworks, principles, and 
questions recommended in this Brief will help educators use data to inform decisionmaking 
and actions related to individual students, groups of students, classes, lesson plans, 
curricula, and other critical aspects of the business of educating students, managing 
classrooms, and operating schools. The Practical Exercise below provides a resource for 
applying these recommendations when dealing with actual issues or illustrative scenarios. 
While these skills can be developed by individual educators and should be a component of 
pre-service education, skill development at a staff level will be enhanced by the delivery 
of professional development curricula around this area of growing importance for faculty 
and staff. Many educators will benefit from collaborating with colleagues when completing 
these data use scenarios. 


Brief 1 Data Use for Educators 


31 






Practical Exercise for Educators: Using What You've Learned 

Educators face a wide range of issues that require that they make decisions each and every day. Following the question- 
driven approach to data use described in Brief I, work individually or brainstorm with colleagues to complete the tables 
below. Be as realistic as possible in your responses, using real data sources, practices, and people in your organization so 
as to make the scenario as useful as possible. 



Framework 

Scenario 

Comments 

Scenario 

An elementary school teacher is preparing an 
instructional plan for the upcoming school year. 

A common occurrence for educators. 

What do 

1 want to 
know? 

Based on this scenario, 1 need to know... 


What data 
might be 
relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 
access the 
relevant 
data? 

Based on the relevant data, i will... 


What skills 
and tools 
do 1 need to 
analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the 
data tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my 
actions have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, I will... 








32 


Forum Guide to Taking Action with Education Data 











Step 5: Step 4: Step 3: Step 2: Step 1: Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 



Scenario 2 


Framework 

Scenario 

Comments 

Scenario 

A high school teacher noticed that a student is 
unable to complete his homework because of 
required skills that seem to be below grade level. 

A common occurrence for educators. 

What do 

1 want to 
know? 

Based on this scenario, 1 need to know... 


What data 
might be 
relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 
access the 
relevant 
data? 

Based on the relevant data, 1 will... 


What skills 
and tools 
do 1 need to 
analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the 
data tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions i will take include... 


What effects 
did my 
actions have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Step 5: Step 4: Step 3: Step 2: Step 1: Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 


Insert Your Own Scenario Table 


Framework 

Scenario 

Comments 

Scenario 



What do 

1 want to 
know? 

Based on this scenario, 1 need to know... 


What data 
might be 
relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 
access the 
relevant 
data? 

Based on the relevant data, i will... 


What skills 
and tools 
do 1 need to 
analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the 
data tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my 
actions have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Brief II: Data Use for School and 
District Leaders 


School and district leaders use data to better meet school and 
district needs 

As a leader in your school or school district, you collect and use data all the time. You might 
need to know student enrollment, class assignments, and staffing plans to order supplies 
for a school campus. Alternatively, maybe you need to make staff assignments based on 
student course taking patterns and teacher qualifications. Perhaps you are responsible for 
tracking daily attendance for a truancy intervention program. Or, your responsibilities 
may include state and federal data collections, grant management reporting, or graduation 
certifications. The bottom line is that school and district leaders access and analyze data 
for a wide range of reasons — many of which are critically important to the day-to-day and 
ongoing operation of buildings, campuses, and districts. If, for example, you can’t access 
contact information for substitute teachers in a timely manner, there may be a district 
administrator who unexpectedly finds himself supervising a classroom full of students 
rather than handling other pressing management and planning duties. 

With the right data, properly applied, you can 

^ track student performance based on course taking history, school characteristics, 
teacher assignments, instructional choices, program participation, and 
demographic background; 

"S identify students who are demonstrating signs of engagement or disengagement 
and how it affects their behavior and academic performance; 

"S study the relationship between teacher characteristics, student growth, and 
teacher effectiveness; 

"S establish, target, and evaluate appropriate staff development programs; 

"S compare budgeted to actual expenditures by account class and create projections 
for future spending; and 

"S collaborate and communicate more effectively with peers in other schools 
and districts. 


With the development 
of more robust data 
systems, ranging 
from building-level 
data dashboards to 
statewide longitudinal 
data systems, school 
and district leaders 
have access to more 
data than ever before. 
The challenge is to use 
the data to improve 
school and district 
management and 
operations rather than 
being overwhelmed 
by the possibilities of 
meaningful data use. 
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Data use by administrators is not new. School and district leaders have always been 
asked to use data to answer a wide range of questions: 

S Instruction : e.g., Do test results and course enrollment patterns suggest that I 
need to hire more math teachers? 

S Classroom Management : e.g., Are there enough computers to meet the state’s 
online testing requirements? 

S Compliance: e.g., Are my building’s third grade teachers dedicating enough 
time to spelling instruction to meet the district’s stated policy on minutes 
on task? 

S Accountability: e.g., Will our high schools meet academic performance targets 
for all student groups? 


The Cycle of Data Use 

How do people use data? 


Wise data use can increase the effectiveness of school and district leaders’ primary 
purpose — operating and managing effective and efficient education organizations. The cycle 
of data use presented in figure 2. 1 has data use to inform action as its primary goal. As such, 
the cycle is based on a framework consisting of five iterative phases (or activities): (1) seek 
information; (2) access/gather data; (3) analyze/interpret data; (4) act; and (5) evaluate. 
Such a framework can serve as the basis for the systematic use of education data in schools 
and districts. 



1. Seek 
Information 


2. Access/ 
Gather Data 


5. Evaluate 


3. Analyze/ 
Interpret Data 


4. Act 


Figure 2.1. The Cycle of Data Use. For a deeper discussion about the cycle of data, see the 
Introduction to the series Forum Guide to Taking Action with Education Data, available at 
http://nces.ed.gov/forum/publications. 
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1. Seek Information 

“What data do I have versus what data do I need?” 

To be effective and efficient, data use should be driven by well-defined questions that are 
intended to improve the operations and management of schools and districts. For example, 
rather than trying to find data to address a very broad question such as, “Flow can I improve 
building efficiencies?” it is likely that a more precisely worded question such as, “Flow have 
school building utility budgets changed over the past five years?” will enable you to find 
more relevant data to inform your planning and action. 

When faced with such a question, ask yourself, “What information do I need (and not 
currently have) to answer this question?” Identifying these gaps will point you toward 
different types of data depending on the information gap you hope to address. For example, 
with the question above about utility costs, in addition to expenses (data you already have), 
you might seek data related to building size, the condition of FIVAC systems, energy savings 
programs at the building level, and even weather conditions during the period of interest. 

2. Access/Gather Data 

“What data do I have and what data can I get?” 

As an education leader, what data do you use from national, state, regional, district, 
or school data systems? Are these data accessible via downloadable data files, grade 
books, reports, spreadsheets, a data warehouse, a student management system’s portals, 
dashboards, analysis instruments, or graphical visualization tools? In addition to the 
information collected by schools and districts (e.g. , attendance data, performance data, 
curriculum data, staffing data), other entities sometimes issue reports about school- and 
district-level fiscal status, facility conditions, long-term planning, etc . 12 Depending on 
your information needs, these may be aggregate level (e.g., grade-level or building- 
level ) or individual-level (e.g., Student First Name), snapshot or longitudinal in nature, 
observational or recorded, and may be in the forms of raw numbers for analysis or 
customized reports that present data specifically for planning purposes. 


Steps to the 

information seeking 

process include 

A Recognizing when 
information is 
needed to inform a 
decision and direct 
an action. 

S Defining a 

meaningful and 
achievable scope 
of concern/unit of 
analysis. 

A Clearly articulating 
the issue at 
hand as a critical 
question. 

A Assessing what 
relevant data are 
already available. 

A Identifying "gaps" in 
currently available 
data. 

'k Identifying potential 
barriers to finding 
the Information and 
ways to overcome 
them. 

'k Seeking additional 
data that are 
accurate, valid, 
reliable, timely, and 
relevant. 


For example, school and district leaders often are charged with determining which high 
school students are on track to graduate with a regular diploma — information that can 
help instructional staff provide assistance to those students who are in a position to possibly 
graduate but who are at risk for not earning a regular diploma with their classmates. If this 
is your job, where do you begin looking for the data you will need to accomplish the task? 

In this scenario, it is likely that the data access/ gathering phase might focus on a host of 
transcript data from your student information system, including grade level, credits earned, 
requirements met, grade point averages, and grades earned in core classes. However, many 


t he Common Core of Data (CCD) is a program of the U.S. Department of Education’s National Center 
for Education Statistics that annually collects fiscal and non-fiscal data about all public schools, public school 
districts, and state education agencies in the United States. CCD data can be accessed and customized using a 
variety of tools available at http:/ / nces.ed.gov/ccd/ . 


Be thoughtful 
consumers of data. 
Realize that some data 
are particularly sensitive 
to change — e.g., district 
enrollment can vary 
simply because of 
the date of collection. 
Verify quality (accuracy, 
timeliness, etc.) before 
using data to take 
action. 


Brief 2 Data Use for School and District Leaders 


39 






Steps to accessing/ 
gathering data include 

S Identifying whether 
the needed data 
already exist. 

S Collecting the data, 
which often means 
gaining access 
to relevant data 
that have already 
been collected by 
someone else. 

S Learning about 
related data, such 
as other data 
sets, applicable 
definitions, 
limitations, timelines, 
formatting, other 
data sets, etc., 
(often referred fo 
as mefadata) to 
ensure that you 
fully undersfand 
the information. 


seasoned administrators know that other factors can strongly influence a student’s success 
during his or her senior year. Other sources of information might focus on a student’s daily 
engagement and disengagement (from class records maintained by teachers), behavioral 
referrals (from discipline systems), participation in a dropout prevention program (from 
grant program records), plans following graduation (from the results of a survey), and 
family stability (from anecdotal information gathered by guidance counselors) . 

Collecting these data often requires engaging numerous data sources, and sometimes the 
challenge is not in finding the data, but understanding them — including, for example, 
recognizing appropriate and spurious relationships between data about student engagement, 
discipline, family stability, and academic persistence (see Step 3: Analyze /Interpret Data). 
Gaining access to data may also be a challenge if you request personally identifiable data. 

You should always consider whether de-identified or aggregate data would meet your need. 

3. Analyze and Interpret Data 

“What do the data suggest?” 

Having access to needed, relevant data is necessary, but not sufficient, for appropriate 
data use. You must also be able to skillfully analyze and take appropriate action based upon 
the information to improve school and district management and operational practices. 
Education leaders who are good data users usually understand 

A the meaning of the data (e.g., “completion rate” is not the same as “graduation 
rate” and “dropout rate” is not 100 percent minus the “graduation rate”); 

A commonly used measures for student performance reporting (e.g., a scale score 
is not the same as a percentile score); 

A basic statistical concepts (e.g., mean, median, mode); 

A trends (e.g., multiple points of data pointing toward the same general trend or 
conclusion are more powerful than one or two data points that aren’t supported 
by other data); 

A common presentation formats (e.g., tabular reports as well as graphs and charts); 
A assumptions that limit the appropriate use of data (e.g., a good reading score 
does not automatically mean that the student is strong in all language arts 
areas); and 

A the value of collaborating with colleagues (e.g., instructional staff relying on data 
staff to improve understanding). 

Failure to understand these aspects of data can result in either ignoring data completely 
when making decisions or, perhaps worse, using data confidently but incorrectly (e.g., 
misinterpretation, over-interpretation, or inappropriate application). 


40 


Forum Guide to Taking Action with Education Data 




Figure 2.2. Analysis includes looking for patterns in the data. Rather than finding one piece 
of data and using it to make a generalization or draw other conclusions that are beyond the 
scope of the single data point, identify multiple data sources that inform the analysis of the 
questions— a process sometimes referred to as “triangulation.” 


Steps to the analysis 
and interpretation 
process include 

S Formatting the 
data in a manner 
that enables sound 
analysis and 
interpretation. 

S Determining any 
limitations on, or 
constraints to, 
data use including 
consideration of 
the data's 
purpose; 
quality; 
bias; and 
timeliness. 

S Reviewing whether 
conclusions are 
logically and 
statistically sound 
and defensible. 


4. Act 

“How can data be acted upon to improve educational processes and outcomes?” 

By definition, “actionable” data refer to information that allows a decision to be made and 
action to be taken. When you, as a school or district leader, are data literate, you are able 
to supplement your education expertise, management skills, and personal experience in 
schools and districts with the right data for the right purposes: to improve the management 
and operations of the education enterprise. 

For example, assessments in many states are used to determine whether academic 
performance goals are being met for accountability purposes. Flowever, those same 
assessments often generate item response data for individual students or groups of students. 
When assessments are designed to provide meaningful feedback at this level, school and 
district leaders can work with instructional staff to identify concepts that students are 
mastering, as well as those in which performance is below proficient. This analysis can 
occur for an individual student, groups of students (e.g., by common subgroups such 
as race/ ethnicity, language learners, and socioeconomic status), and at the classroom 
and grade levels. Trends in these data can then be used to assess the relative strengths 
and weaknesses of the curriculum, various instructional strategies, and even individual 
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teachers in need of professional development or who might serve as expert resources for 
others. Based on these findings, you may even change or supplement the curriculum and 
instructional support tools as necessary to better meet the specific needs of your schools, 
staff, and students. 

5. Evaluate 

“Has the issue changed for the better?” 

Actionable data are powerful tools that often lead to change. The question is whether 
this change is for the better, and whether the situation has improved sufficiently to have 
adequately addressed the original concern. For example, if you apply the data use cycle 
to evaluate the district’s math curriculum, it is critical to know whether math skills and 
performance are improving across the student population (as well as within particular 
subgroups). In addition to determining whether there is academic improvement, you 
may also wish to assess whether there is sufficient progress — that is, whether the student 
population is achieving expected growth targets rather than just improving basic skills 
that are still not satisfactory. As such, determining whether the original issue has been 
adequately addressed is an important component of the data use process. This type of 
evaluation often includes consideration of the following types of questions: 

S Flow effectively has the initial issue been resolved? 

What new concerns have arisen? 

'E Which factors are understood sufficiently and which warrant 
additional investigation? 

'E Flave new data gaps (needs) been identified? 

If the issue has been effectively addressed by data-informed action, then you may wish to 
share your experience with colleagues and encourage them to use data to deal with similar 
problems. If the issue hasn’t been fully resolved, the cycle of data use repeats itself (see Step 
1 : Seek Information). 

Critical Data Use Skills 

Data are becoming increasingly relevant to the core management duties of school and 
district leaders. Some fundamental knowledge, skills, and abilities needed to effectively 
analyze and interpret data — and, thus, use education data effectively — include 

Understand what the data mean. Is program participation the same as 
program completion? Do program participation data in a table refer to a point 
in time during a particular academic year or an average over several years? Data 
users have difficulty knowing what the data really mean unless they understand 
data definitions, how the data are collected and formatted, and other usage 
guidance and limitations. Data about data are often referred to as “metadata.” 13 


13 Forum Guide to Metadata:The Meaning Behind Education Data. Available at 
http: / / nces. ed.gov/Jorum /pub_2009805.asp. 
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What Do the Data Mean? 

The Graduation Rate in many states includes only those students receiving regular, 
standard, endorsed, or advanced diplomas. In contrast, the Completion Rate is generally 
used to measure graduates as well as students who receive a high school equivalency 
certificate, certificate of completion, or attendance certificate. The meaning of these 
similar data terms clearly depends on how the terms “graduate” and “completer” 
are defined, and anyone using the information would benefit from clear and accurate 
definitions for the terms. 


s View data in an appropriate context. Data users should avoid jumping to 
a conclusion based on a single piece of data without being sure they understand 
the context of the data. When you evaluate data, think about complementary 
information that paints a richer picture of the education experience. For 
example, if you notice that class-level assessment scores don’t fully measure the 
effectiveness of a teacher, you might argue for a teacher evaluation system that 
incorporates other factors such as classroom observations, course load, co- 
teachers, and student characteristics. The point is that the teaching and learning 
occur within the broader context of a school and community setting, and the 
data that might help to inform the situation should reflect that broader context. 


Why Does Context Matter? 

Temperature is a measurement most people think they understand, but to correctly 
interpret it as information requires additional context. 

x' What is the scale? Degrees Celsius or Fahrenheit? 
x' Is it the high or low temperature for the day? 
x' Is it the current temperature, yesterday’s temperature, or tomorrow’s 
predicted temperature? 

This context is critical to using the piece of data as information. In its absence, “28” could 
mean 28° Fahrenheit as the day’s high temperature (brrr, is it cold) or 28° Celsius (the 
equivalent of 82 ° Fahrenheit) as tomorrow’s predicted low temperature (wow, will it ever 
be a scorcher). 


s Recognize the nature of the collection tool. Most data collection tools are 
designed to generate data for a specific purpose. For example, some assessments 
measure skill development across a broad spectrum, others evaluate content 
mastery relative to a standard criteria, and still others are used to rank student 
performance relative to other students. Similarly, some reading assessments 
are especially sensitive to skill development in emerging readers, but less 
effective indicators of progress being made by strong readers. Thus, each type of 
assessment is constructed for a different purpose, which should inform how the 
results are used for school improvement and planning. 
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s Identify bias in the collection source. Be wary of data sources that may 
have a bias. For example, what if someone shared data from a study that showed 
that buying a piece of instructional software would substantially raise your 
district’s reading scores? As a school or district leader, would you view the data 
differently if it originated in an objective source (e.g., another local education 
agency using an independent evaluator) as opposed to being published by the 
company that sells the product? 


Timely data are useful 
data. Acting on ten- 
year-old data is not 
likely to improve future 
educational outcomes. 
Make sure that the 
data you are using 
are timely enough to 
describe current status 
and conditions. 


Do I Really Need to Understand Error? 

The effects of error measures can have substantial impacts on the actions of teachers. 

For example, depending on the uncertainty of the measures, students hoveringjust above 
and just below proficiency on state assessments may be equally at risk to score below 
proficient on future assessments. 


"S Assess the quality of the data. Data quality will limit the effectiveness 
of actions based on data. In other words, are your data accurate, reliable, and 
timely? For example, an end-of-year science assessment that only tests content 
from the last eight weeks of the academic calendar is not likely to be a valid 
measure of student learning for the entire year. Similarly, if a survey were to 
be administered to similar groups of students on different days, how likely is it 
that respondents would reliably provide the same responses? If it is not likely to 
generate similar findings, the instrument may not be reliable and the quality of 
the data is uncertain. 14 


V Appreciate the assumptions and limitations of data. Oftentimes, 

information about data meaning, format, and usage are available as metadata 
(data about data) . For example, the results of a norm-referenced test are 
designed to reflect a normal curve, meaning that such an assessment is usually 
not a good instrument for evaluating the success of changes to the instructional 
program. Good data use requires an understanding of the assumptions and 
limitations of each piece of data. As a school or district leader who needs to act 
on data, you should expect to collaborate with data managers, data stewards, 
or other staff charged by the organization to understand the intricacies of data 
elements, code sets, formulae, definitions, formats, and meaning. 15 


V Comprehend the significance of error and/or uncertainty in data. 

While calculating error requires considerable statistical training, even a novice 
data user can improve the likelihood of sound usage by taking a few moments to 
consider the implications of error measures. As a school or district leader, you 


14 Forum Curriculum for Improving Education Data: A Resource Jor Local Education Agencies, available at 
http: / / nces. ed.gov/Jorum / pub_2007 808 .asp. 

15 Forum Guide to Building a Culture cj Quality Data: A School and District Resource, available at 
http: / / nces. ed.gov/Jorum / pub_2005801 .asp. 
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should understand that confidence intervals and other methods of 
describing uncertainty provide important information about the 
limits of data accuracy. 

Discern meaning in tabular and graphical data. Because of 

the widespread use of tabular and graphical data in school, district, 
and state reporting, you will want to become comfortable with, 
and skilled at, analyzing data in these formats. Interpreting tabular 
data requires that you pay close attention to column and row titles, 
as well as footnotes and other appended materials, which often 
provide a context in which to interpret meaning. Reading a graph 
can be fairly straightforward, but understanding the relationship 
between the x- and y-axis requires some training. For example, the 
commonly cited statistical maxim “correlation is not causation” is 
particularly applicable to graphical data — just because there is a 
visual relationship between points on a graph does not mean that 
one variable caused the other to occur. 

Know how to effectively question data/report authors. 

Sometimes the fiscal data you receive from the state education 
agency is wrong. Other times, research reports draw conclusions 
that don’t seem obvious from the data. How does a school or 
district leader deal with inaccurate, misleading, or poorly presented 
data? Unlike some data skills that are heavily quantitative or 
analytical in nature, effectively questioning a data source is mostly 
a function of effort and confidence. When a data report or research 
finding appears to be inaccurate, misleading, or unclear to you, a 
practicing administrator, then it is entirely appropriate to contact the 
source of the data to politely and professionally request clarification 
or correction. In addition to addressing your immediate concerns, 
you may help the author make the data more accurate, more 
understandable, and more actionable for the entire education community. 
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Figure 2.3. Because of the widespread 
use of tabular and graphical data, data 
users should become comfortable with, 
and even skilled at, analyzing data in 
tables and figures. 


Training and Skill Development 

Ideally, administrator preparation programs would begin the process of developing data 
use skills. Current professionals might receive informal support through peer-to-peer 
study and professional learning communities. A great deal of information concerning 
the processes and techniques that support good data use can also be found online by 
enterprising individuals who understand the value of strong data use skills. More formally, 
professional development opportunities might include training that emphasizes topics such 
as recognizing whether data are within acceptable ranges, identifying patterns or trends 
in the data, and applying this information to solve real-world problems. Similarly, trainers 
with data expertise can develop a staff’s quantitative skills, such as appreciating the meaning 
of error measures; calculating the amount of change a variable presented in charts, tables, 
and graphs; and interpreting the significance of statistical summaries of large data. 
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A Question-Driven Approach to Data Use 

Using the skills described in the previous section, school and district leaders can rely on a 
question-driven approach to data use for nearly any scenario that requires making a data- 
informed decision and taking data-informed action. 

A What do I want to know? 

A What data might be relevant? 

A How will I access the relevant data? 

A What skills and tools do I need to analyze the data? 

A What do the data tell me? 

A What are my conclusions? 

A What will I do? 

A What effects did my actions have? 

A What are the next steps? 


Table 1 and table 2 illustrate how these questions can be applied to example scenarios 
commonly seen by school and district leaders. 


Table 1. An illustration of a question-driven approach to data use for various scenarios facing school and district leaders 
(example l) 16 


<D 

U 

Framework 

Scenario 

Comments 

U 

<D 

Scenario 

A school leader is planning course schedules for the 

A common occurrence for school and district 

ZD 


upcoming school year. 

leaders. Good school and district leaders often 

O 



customize previously successful course schedules 

o 

o 



to meet the learning needs of incoming students. 


What do 1 want to 

Based on this scenario, 1 need to know... 

Subsequent questions include: 


know? 


What is the academic and behavioral profile of 

o 


Are student needs and available resources 

individual students? 



sufficiently similar to last year to warrant not 

What is the aggregate profile of the grade/ 

<1> 


changing the instructional plan? 

school? 

oo 



If the plan does need to be modified, what 




student, staff, and school needs must be 

<D 



accommodated? 

<5 



What modifications should 1 make to last year's 




successful course offering schedule? 


16 In some cases, you may collapse these steps in your day-to-day application of this framework. For example, if you have taught a student for 
several months, you might already know the students profile. In other instances, you may wish to rely on this framework more precisely as you 
formally analyze the data. 
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Step 3: Step 2: 

Analyze/Interpret Data Access/Gather Data 


A, Ifl 


Table 1. An illustration of a question-driven approach to data use for various scenarios facing school and district leaders 
(continued) 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 

E Curriculum data 

E Has the curriculum been changed? 

E Academic performance data 

E Enrollment projection data 

E Previous classroom experience data 

E Which courses should be offered to keep 
students on pace/level? 

E Program participation data 

E Have individual students been identified 
as having special learning needs? 

E Budget data 

E Do we have more or fewer resources? 

E Staff data 

□ Building data 

E Fiscal data 

□ Community input 

Although there may be a wealth of data that 
could be evaluated, the scenario may warrant 
only a small subset of data for analysis. 

How will 1 access 

Based on the relevant data, 1 will... 

You may already have access to these data 

the relevant 


sources; other needs may require gathering data 

data? 

E consult colleagues about observational 
and anecdotal data 

you do not already have. 


E reflect on my recent experiences with the 

The process for accessing these data may vary 


grade/school 

□ consult a student's portfolio to review past 
work 

□ query the student information system for 
interim benchmark data 

E query the student information system for 
summative data 

□ use my gradebook to review the homework 
data 

E invite parents and community members to 
share opinions about needs and 
expectations 

by organization. 

What skills and 

Based on gaining access to these data, 1 will need to 

Don't be too quick to form an opinion based 

tools do 1 need 

be able to... 

on one or even a few pieces of data. Use 

to analyze the 


techniques such as triangulation to assimilate 

data? 

E understand the context and how the data 

and evaluate the body of data in order to 


are presented 

E organize the data in a way that applies to 
this situation 

E relate, combine, and balance information 
from different sources 

E assess the data quality and limitations 

develop conclusions. 

What do the data 

Based on this analysis, the data suggest... 

Are there gaps in the data? In other words, are 

tell me? 


data that would be helpful to evaluate missing 


E student/school needs comparable to 

or otherwise unavailable? If so, who needs to be 


previous year 

told this information so that it can be corrected in 


□ student/school needs not comparable to 
previous year 

□ resources/budgets comparable to previous 
year 

E resources/budgets not comparable to 
previous year 

the future? 
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Step 5: Step 4: 

Evaluate Act on the Data 


Ills 


Table 1. An illustration of a question-driven approach to data use for various scenarios facing school and district leaders 
(continued) 


What are my 
conclusions? 

Based on this interpretation, 1 conclude that... 

□ it is necessary to modify the course 
schedule based on student/school needs 

0 it is not necessary to modify the course 
schedule based on student/school needs 

□ it is necessary to modify the course 
schedule based on decrease in available 

resources 

0 it is necessary to modify the course 

schedule based on decrease in available 

resources 

If it is not an issue that requires action, then 
it is the end of the story. If it is an issue that 
requires action, then more information becomes 
necessary. 

What will 1 do? 

Based on this conclusion, actions 1 will take include... 

□ find more information (check other data 
sources) 

0 increase class size in order to avoid 
reducing course offerings 

□ request changes in staffing to improve 
capacity to teach new courses 

□ decrease the number of courses offered 

0 ask for more resources so that course 

schedules reflect student/school need 

Multiple actions may be selected. For example, 
the school or district leader may choose 
to increase class size while simultaneously 
requesting additional resources so that class sizes 
could remain unchanged. 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 

0 the revised course offering is appropriate 
for the needs and resources 
□ the revised course offering is not 

appropriate for the needs and resources 

At a later point in time, 1 can assess whether my 
action (based on data use) was successful. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

0 monitor whether resource availability 
changes 

0 observe whether grade/school populations 
continue to demonstrate sufficient 
academic progress 

0 evaluate whether subsequent questions 
have arisen and, if necessary, using this 
question-driven approach to data use to 
advance the inquiry 

If the situation is not completely resolved, the 
school or district leader may wish to identify what 
specific issues still need to be addressed and 
return to the beginning of the data use cycle 
to seek new information that will inform future 
action. 
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Table 2. An illustration of a question-driven approach to data use for various scenarios facing school and district leaders 
(example 2) 16 


<D 

U 

>S 

Framework 

Scenario 

Comments 

O 

<D 

Scenario 

A district leader is evaluating teachers. 

A common occurrence for school and district 

ID 




leaders. Good school and district leaders often 

o 




consider a range of factors when assessing staff 

Q 




performance. 


What do 1 want to 

Based on this scenario, 1 need to know... 

Subsequent questions include: 


know? 



What is the academic and behavioral profile of 

^ C 


How are each of my teachers performing? 

individual students in the class/school? 

a v 




Are there particular academic and behavioral 

to <D 
to 




situations to accommodate? 


What data might 

Based on the information 1 need, possibilities 

Although there may be a wealth of data that 


be relevant? 

include. 


could be evaluated, the scenario may warrant 
only a small subset of data for analysis. 



E 

student performance (individual and class 
level assessment scores) 




E 

classroom observations 




□ 

peer review 




E 

teacher attendance 




E 

past evaluations (e.g., accomplishing prior 
goals and/or remediation) 




E 

course load, course type considerations 




□ 

other assignment-related factors 




□ 

class/student characteristics 




□ 

participation in professional development 


O 

o 

o 


E 

student/parent reviews and/or complaints 


.. 

How will 1 access 

Based on the relevant data, 1 will... 

You may already have access to these data 

a o 

the relevant 



sources; other needs may require gathering data 

a > o 

<D 

U 

data? 

E 

consult colleagues about observational 
and anecdotal data 

you do not already have. 


□ 

reflect on my recent experiences with the 

The process for accessing these data may vary 

< 


E 

teacher/class/grade/school 

review each teacher's portfolio to review 

past work 

by organization. 



E 

review each teacher's HR file 




E 

review each teacher's professional 
development records 




□ 

query the student information system for 
interim benchmark data 




E 

query the student information system for 
summative data 




□ 

use my gradebook to review the homework 
data 




E 

invite parents and community members to 
share opinions 



16 In some cases, you may collapse these steps in your day-to-day application of this framework. For example, if you have taught a student for 
several months, you might already know the student’s profile. In other instances, you may wish to rely on this framework more precisely as you 
formally analyze the data. 
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Step 5: Step 4: Step 3: 

Evaluate Act on the Data Analyze/Interpret Data 



Table 2. An illustration of a question-driven approach to data use for various scenarios facing school and district leaders 
(continued) 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 

0 understand the context and how the data 
are presented 

0 organize the data in a way that applies to 
this situation 

0 relate, combine, and balance information 
from different sources 

0 assess the data quality and limitations 

Don't be too quick to form an opinion based 
on one or even a few pieces of data. Use 
techniques such as triangulation to assimilate and 
evaluate the body of data in order to develop 
conclusions. 

What do the data 
tell me? 

Based on this analysis, the data suggest... 

0 generally adequate teacher performance 

□ a record of variable performance 

□ unacceptable teacher performance 

□ unclear results 

Are there gaps in the data? In other words, are 
data that would be helpful to evaluate missing 
or otherwise unavailable? If so, who needs to be 
told this information so that it can be corrected? 

What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

0 the teachers performed within desired 
expectations 

□ the teachers did not perform within desired 
expectations 

Teachers are evaluated individually and results 
can be aggregated to inform broad planning 
tasks. 

What will 1 do? 

Based on this conclusion, actions 1 will take include... 

0 find more information (check other data 
sources) 

0 identify specific topics or skills that may 
warrant special emphasis in professional 
development 

□ modify the teacher’s assignment 

□ recommend the termination of the 
employee's teaching contract 

Although the performance of a teacher may 
be acceptable, it is possible that participation in 
targeted professional development could further 
improve performance. 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 

0 the teachers are performing adequately 
□ the teachers are not improving 

At a later point in time, 1 can assess whether my 
action (based on data use) was successful. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

0 continue to monitor the situation 

0 observe whether the teachers continue to 
demonstrate adequate performance 

0 evaluate whether subsequent questions 
have arisen and, if necessary, using this 
question-driven approach to data use to 
advance the inquiry 

If the situation is not completely resolved, the 
school or district leader may wish to identify what 
specific issues still need to be addressed and 
return to the beginning of the data use cycle 
to seek new information that will inform future 
action. 
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Conclusions 

High quality and timely data are powerful tools for improving the management and 
operation of the entire education enterprise. Thoughtful application of the frameworks, 
principles, and questions recommended in this Brief will help school and district leaders 
use data to inform decisionmaking and actions related to individual students, groups of 
students, classes, buildings, campuses, districts, and other critical aspects of the business 
of educating students, managing classroom, and operating schools. The Practical Exercise 
below provides a resource for applying these recommendations when dealing with actual 
issues or illustrative scenarios. While these skills can be developed by individuals, skill 
development at a staff level will be enhanced by the delivery of professional development 
curricula around this area of growing importance for school and district leaders. Many 
school and district leaders will benefit from collaborating with colleagues when completing 
these data use scenarios. 
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Practical Exercise for School and District Leaders: Using What You've Learned 

School and district leaders face a wide range of issues that require that they make decisions each and every day. Following 
the question-driven approach to data use described above, work individually or brainstorm with colleagues to complete 
the tables below. Be as realistic as possible in your responses— using real data sources, practices, and people in your 
organization so as to make the scenario as useful as possible. 


<1) 

£» <D 

Framework 

Scenario 

Comments 

5 £ 

D <J 

O 

Scenario 

An elementary school principal is making staffing 
assignments for the upcoming school year. 

A common occurrence for educators. 

Step 1: 

Seek Info 

What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


o 

o 

o 

.. <D 

CN £ 

a o 

<D CD 

What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


55 's- 

(A 

a> 

u 

o 

< 

How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


o 

D 

Q 

a> 

^ a 

What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


Step 

Analyze/Inter 

What do the data 
tell me? 

Based on this analysis, the data suggest... 


o 

o 

What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


Step 4: 
Act on the C 

What will 1 do? 

Based on this conclusion, actions 1 will take include... 


.. a> 

10 o 
a d 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


</> > 

UJ 

What are the 
next steps? 

Based on my evaluation, 1 will... 
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Step 5: Step 4: Step 3: Step 2: Step 1: Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 


71 A 


Scenario 2 


Framework 

Scenario 

Comments 

Scenario 

A high school principal needs to allocate resources 
to different support programs that serve a wide 
range of fluctuating student populations and needs. 

A common occurrence for educators. 

What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the data 
tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Step 5: Step 4: Step 3: Step 2: Step 1: Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 


N. 

if 



Insert Your Own Scenario Table 


Framework 

Scenario 

Comments 

Scenario 



What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the data 
tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Brief III: Data Use for State 
Program Staff 

State program staff use data to better manage state and 
federal programs 

As a program area administrator at a state education agency, you are responsible for 
allocating funds for program services, identifying best practices for program delivery, and 
ensuring compliance with relevant state and federal program requirements. As such, it 
is likely that you collect and use data all the time. Whether you work in early childhood, 
special education, career and technical education, or any of a wide range of program 
areas, you probably need to know the number and demographic characteristics of eligible 
students from across the state, as well as the number and characteristics of those children 
actually enrolled and receiving program services. You may also need to develop business 
rules to maximize program effectiveness and efficiency, or perhaps your responsibilities 
include reporting descriptive data about the program sites to state or federal oversight 
bodies. 

The bottom line is that state program staff access and analyze data for a wide range of 
reasons — many of which are critically important to the day-to-day and ongoing operation 
and management of education programs and support services across a state. If, for example, 
you can’t determine how much money was allocated to a program site, state auditors might 
submit an unfavorable report to the legislative body that funds the program — meaning that 
some eligible participants may not be able to participate in future years. 

With the recent development of more robust data systems, ranging from program-level 
data dashboards to statewide longitudinal data systems, state program staff have access to 
more data than ever before. With the right data, properly applied, you can 

^ track student eligibility and participation at the school, building, or site level; 

"S identify program participants based on demographic backgrounds and other 
common subgroups; 

"S evaluate the effectiveness of a program (e.g., relative to program goals 
and milestones); 

"S compare budgeted to actual expenditures by account class and create projections 
for future spending; and 

communicate more effectively with colleagues in other states, programs, schools, 
districts, and sites. 


State program staff 
focus on a wide range 
of service areas, 
including 

S early childhood 
education; 

S career and 

technical education; 
S migrant education; 
S special education; 

S homeless services; 

school finance; 

S school construction; 
S teacher certification 
and quality; 

S college and career 
readiness; 

S federal title 
programs; and 
S child nutrition and 
wellness. 
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Data use by state program staff is not new. State program specialists have always been 
asked to use data to answer a wide range of questions: 

•/ Compliance : e.g., Are only students who are eligible for program participation 
receiving services? 

S Accountability: e.g., Do I have evidence that program funds have been 
spent appropriately? 

S Program effectiveness: e.g., Are program participants demonstrating improved 
academic performance and graduation rates? 

S Technical assistance: e.g., Do I understand the characteristics of the program 
site and participant population well enough to provide appropriate 
technical assistance? 


The Cycle of Data Use 

How do people use data? 


Wise data use can increase the effectiveness of state program staff’s primary purpose: 
providing services to eligible populations. The cycle of data use presented in figure 3.1 has 
data use to inform action as its primary goal. As such, it is based on a framework consisting 
of five iterative phases (or activities): (1) seek information; (2) access/ gather data; (3) 
analyze /interpret data; (4) act; and (5) evaluate. Such a framework can serve as the basis for 
the systematic use of education data in state education agencies. 



1. Seek 
Information 


2. Access/ 
Gather Data 


5. Evaluate 


3. Analyze/ 
Interpret Data 


4. Act 


Figure 3.1. The Cycle of Data Use. For a deeper discussion about the cycle of data, see the 
Introduction to the series Forum Guide to Taking Action with Education Data, available at 
http://nces.ed.gov/forum/publications. 
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1. Seek Information 

“What data do I have versus what data do I need?” 

To be effective and efficient, data use should be driven by well-defined questions that 
are intended to improve the operations and management of state program services. For 
example, rather than trying to find data to address a very broad question such as “Are our 
programs working?”, it is likely that a more precisely worded question such as “Is the high 
school graduation rate increasing for students receiving program services?” will enable you 
to find more relevant data to inform planning and action. 

When faced with such a question, ask yourself, “What information do I need (and not 
currently have) to answer this question?” Identifying these gaps will point you toward 
different types of data depending on the information gap you hope to address. For example, 
with the question above about program participants completing high school, in addition 
to current program rosters (data you already have), you might seek historical data about 
program rosters, graduation rates, and completion rates during the periods of interest. 

2. Access/Gather Data 

“What data do I have and what data can I get?” 

As a state program administrator, what data do you use from national, state, regional, 
district, or school data systems? Are these data accessible via downloadable data files, 
reports, spreadsheets, a data warehouse, or a business intelligence system with portals, 
dashboards, analysis instruments, and graphical visualization tools? In addition to the 
information collected by states and districts (e.g. , student demographic data, attendance 
data, and performance data), other entities sometimes issue reports about state- and 
district-level fiscal status, facility conditions, long-term planning, etc. 18 Depending on your 
information needs, these may be aggregate level (e.g., grade-level or building-level ) or 
individual-level (e.g., Student First Name), snapshot or longitudinal in nature, observational 
or recorded, and may be in the form of raw numbers for analysis or customized reports that 
present data specifically for planning purposes. Other pertinent information might describe 
a student’s participation in related programs or family participation in social services, 
human services, and community health programs. Accessing these data often requires 
engaging numerous data sources, and sometimes the challenge is not in finding the data, 
but understanding them — including, for example, recognizing appropriate and spurious 
relationships between data about program eligibility, participation, completion, family 
stability, and academic persistence (see Step 3: Analyze /Interpret Data). Gaining access to 
data may also be a challenge if you request personally identifiable data. You should always 
consider whether de-identified or aggregate data would meet your need. 


18 The Common Core of Data (CCD) is a program of the U.S. Department of Education’s National Center 
for Education Statistics that annually collects fiscal and non-fiscal data about all public schools, public school 
districts and state education agencies in the United States. CCD data can be accessed and customized using a 
variety of tools available at http:/ /nces. ed.gov/ccd/ . 


Be thoughtful 
consumers of data. 

Realize that some data 
are particularly sensitive 
to change — e.g., class 
enrollment can vary 
simply because of 
the date of collection. 
Verify quality (accuracy, 
timeliness, etc.) before 
using data to take 
action. 


Steps to the 

Information seeking 

process Include 

V Recognizing when 
information is 
needed to Inform a 
decision and direct 
an action. 

V Defining a 
meaningful and 
achievable scope 
of concern/unit of 
analysis. 

V Clearly articulating 
the issue at 
hand as a critical 
question. 

V Assessing what 
relevant data are 
already available. 

■A Identifying "gaps" In 
currently available 
data. 

Identifying potenfial 
barriers to finding 
the information and 
ways to overcome 
them. 

S Seeking additional 
data that are 
accurate, valid, 
reliable, timely, and 
relevant. 
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3. Analyze and Interpret Data 

“What do the data suggest?” 


Steps to accessing/ 
gathering data include 

S Identifying whether 
the needed data 
already exist. 

S Collecting the data, 
which often means 
gaining access 
to relevant data 
that have already 
been collected by 
someone else. 

S Learning about 
related data, such 
as other data 
sets, applicable 
definitions, 
limitations, timelines, 
formatting, other 
data sets, etc., 
(often referred to 
as metadata) to 
ensure that you 
fully understand 
the information. 


Having access to needed, relevant data is necessary, but not sufficient, for appropriate data 
use. You must also be able to skillfully analyze and take appropriate action based upon the 
information to improve program management and operations. Program administrators who 
are good data users usually understand 

A the meaning of the data (e.g., “completion rate” is not the same as “graduation 
rate” and “dropout rate” is not 100 percent minus the “graduation rate”); 

■A commonly used measures for district, school, student, and subgroup 

performance reporting (e.g., a scale score is not the same as a percentile score); 

^ basic statistical concepts (e.g., mean, median, and mode); 

•A trends (e.g. , multiple points of data pointing toward die same general trend or conclusion 
are more powerful than one or two data points that aren’t supported by other data); 

■A common presentation formats (e.g., tabular reports as well as graphs and charts); 

•A assumptions that limit the appropriate use of data (e.g., data from a point in time 
collection date aren’t necessarily representative of longer periods of time); and 

A the value of collaborating with colleagues (e.g., state staff relying on data staff to 
improve understanding). 



Figure 3.2. Analysis includes looking for patterns in the data. Rather than finding one piece 
of data and using it to make a generalization or draw other conclusions that are beyond the 
scope of the single data point, identify multiple data sources that inform the analysis of the 
questions— a process sometimes referred to as “triangulation.” 
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4. Act 

“How can data be acted upon to improve educational processes and outcomes?” 

By definition, “actionable” data refer to information that allows a decision to be made and 
action to be taken. When you, as a state program administrator, are data literate, you are 
able to supplement your program expertise, management skills, and personal experience at 
the state agency with the right data for the right purposes: to improve the management and 
operations of program services. 

For example, some states rely on career “pathway” programs to help allocate resources to 
targeted pre -professional areas based on employment opportunities and service demands 
in various regions of a state. Perhaps there aren’t enough teacher aides in one region 
of the state, and not enough people with technical skills in another region. If this is the 
case, pathway programs aim to respond to these needs and opportunities by offering 
targeted secondary school coursework that helps prepare students for these occupations. 

By identifying employment needs and analyzing student and program data (employee 
capabilities), program staff are able to act by creating new pathway programs that meet the 
employment demands of individual regions across the state. Education resources are used 
to help both a state’s employers and future employees (students who participate in pathway 
programs) because of the ability of program staff to interpret and act on data. 

5. Evaluate 

“Has the issue changed for the better?” 

Actionable data are powerful tools that often lead to change. The question is whether 
this change is for the better, and whether the situation has improved sufficiently to have 
adequately addressed the original concern. For example, if you apply the data use cycle to 
evaluate the academic performance of program participants, it is critical to know whether 
performance measures are improving across the entire student population or only in the 
program participation subgroup. In addition to determining whether there is academic 
improvement, you may also wish to assess whether there is sufficient progress — that is, 
whether the student population is achieving expected growth targets rather than just 
improving basic skills that are still not satisfactory. As such, determining whether the 
original issue has been adequately addressed is an important component of the data use 
process. This type of evaluation often includes consideration of the following types of 
questions: 

S How effectively has the initial issue been resolved? 

What new concerns have arisen? 

Which factors are understood sufficiently and which warrant 
additional investigation? 

A Have new data gaps (needs) been identified? 


Steps to the analysis 
and interpretation 
process include 

S Formatting the 
data in a manner 
that enables sound 
analysis and 
interpretation. 

S Determining any 
limitations on, or 
constraints to, 
data use including 
consideration of 
the data's 
purpose; 
quality; 
bias; and 
timeliness. 

S Reviewing whether 
conclusions are 
logically and 
statistically sound 
and defensible. 
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If the issue has been effectively addressed by data-informed action, then you may wish to 
share your experience with colleagues and encourage them to use data to deal with similar 
problems. If the issue hasn’t been fully resolved, the cycle of data use repeats itself (see Step 
1 : Seek Information). 

Critical Data Use Skills 

Data are becoming increasingly relevant to the core management duties of state program 
staff. Some fundamental knowledge, skills, and abilities needed to effectively analyze and 
interpret data — and thus use education data effectively — include 

Understand what the data mean. Is program participation the same as 
program completion? Do program participation data in a table refer to a point 
in time during a particular academic year or an average over several years? Data 
users have difficulty knowing what the data really mean unless they understand 
data definitions, how the data are collected and formatted, and other usage 
guidance and limitations. Data about data are often referred to as “metadata.” 19 
More broadly, it is important that you learn about the types of information 
contained in the data systems you are able to access, keeping in mind that it 
might be possible to link data points from multiple disparate systems to address 
your information needs. 


What Do the Data Mean? 

The Graduation Rate in many states includes only those students receiving regular, 
standard, endorsed, or advanced diplomas. In contrast, the Completion Rate is generally 
used to measure graduates as well as students who receive a high school equivalency 
certificate, certificate of completion, or attendance certificate. The meaning of these 
similar data terms clearly depends on how the terms “graduate” and “completer” 
are defined, and anyone using the information would benefit from clear and accurate 
definitions for the terms. 


V View data in an appropriate context. Data users should avoid jumping to 
a conclusion based on a single piece of data without being sure they understand 
the context of the data. When you evaluate data, think about complementary 
information that paints a richer picture of program services and the education 
experience. For example, if you notice that assessment scores don’t fully measure 
the effectiveness of a program, you might argue for an evaluation system tint 
incorporates other factors such as participant dropout and persistence rates, 
mobility rates, and employment rates. The point is that the program participation 
occurs within a broader context and data that might help to inform decisions and 
actions should reflect that broader context. 


19 Forum Guide to Metadata:The Meaning Behind Education Data, available at 
http: / / nces. ed.gov/Jorum / pub_2009805 .asp. 
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Why is context critical? 

A graduation rate of 40 percent might seem low, but what if it was at a school that 
specialized in trying to retain students who were at-risk for dropping out? In such a 
case, context is critically important for interpreting a 40 percent graduation rate as 
tremendously successful in such a setting. 


V Recognize the nature of the collection tool. Most data collection 

tools are designed to generate data for a specific purpose. For example, some 
assessments measure skill development across a broad spectrum, while others 
evaluate content mastery relative to a standard criteria. Still others are used 
to rank school or district performance relative to other schools and districts. 
Similarly, some collections gather Days of Attendance and Days in Membership, 
while other collection tools can identify whether a student is present on any 
given day. Both might enable calculation of an attendance rate, but since they are 
constructed for different purposes, they may yield different results. The purpose 
and nature of a data collection should inform how the data are used to measure 
program effectiveness. 


s Identify bias in the collection source. Be wary of data sources that may have 
a bias. For example, as a state program administrator, would you view data about 
a particular service model differently if the data originated in an accountability 
report from a paid service provider as opposed to an independent auditor? 

^ Assess the quality Of the data. Data quality will limit the effectiveness 
of actions based on data. In other words, are your data accurate, reliable, and 
timely? For example, data quality may suffer during the first year of collecting 
a new data element while respondents learn how to interpret definitions and 
code responses. Similarly, quality can vary depending on whether data are self- 
reported or generated by an audit. 20 


Timely data are useful 
data. Acting on ten- 
year-old data is not 
likely to Improve future 
educational outcomes. 
Make sure that the 
data you are using 
are timely enough to 
describe current status 
and conditions. 


V Appreciate the assumptions and limitations of data. Oftentimes, 

information about data meaning, format, and usage are available as metadata 
(data about data). For example, understanding the criteria for selecting “peer 
programs” is critical when evaluating a comparison of one program to other 
programs assumed to operate in “comparable” settings. Are they similar because 
of size, funding, demographic characteristics, or other reasons? Good data use 
requires an understanding of the assumptions and limitations of each piece of 
data. As a state program administrator who needs to act on data, you should 
expect to collaborate with data managers, data stewards, or other staff charged 


20 Forum Curriculum jor Improving Education Data: A Resource jor Local Education Agencies, available at 
http: / / nces.ed.gov / jorum / pub _2007 808 .asp. 
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by the organization to understand the intricacies of data elements, code sets, 
formulae, definitions, formats, and meaning . 21 



Figure 3.3. Because of the widespread 
use of tabular and graphical data, data 
users should become comfortable with, 
and even skilled at, analyzing data in 
tables and figures. 


V Comprehend the significance of error and/or uncertainty in 

data. While calculating error requires considerable statistical training, even 
a novice data user can improve the likelihood of sound usage by taking a few 
moments to consider the implications of error measures. As a state program 
administrator, you should understand that confidence intervals and other 
methods of describing uncertainty provide important information about the 
limits of data accuracy. For example, students “on the bubble” (that is hovering 
just above and just below proficiency on state assessments) may be equally 
at risk statistically to score below proficient on future assessments — likely 
warranting comparable support services even though one group is currently 
viewed to be proficient. 

V Discern meaning in tabular and graphical data. Because of the 

widespread use of tabular and graphical data in school, district, and state 
reporting, you will want to become comfortable with, and skilled at, analyzing 
data in these formats. Interpreting tabular data requires that you pay close 
attention to column and row titles as well as footnotes and other appended 
materials that often provide a context in which to interpret meaning. Reading 
a graph can be fairly straightforward, but understanding the meaning between 
the x- and y-axis requires some training. For example, the commonly cited 
statistical maxim “correlation is not causation” is particularly applicable to 
graphical data — just because there is a visual relationship between points on a 
graph does not mean that one variable caused the other to occur. 

V Know how to effectively question data/report authors. 

Sometimes the fiscal data reported by a newspaper of special interest group 
are misleading. Other times, research reports draw conclusions that don’t 
seem obvious from the data. Flow does a state program administrator deal 
with inaccurate, misleading, or poorly presented data? Unlike some data skills 
that are heavily quantitative or analytical in nature, effectively questioning a 
data source is mostly a function of effort and confidence. When a data report 
or research finding appears to be inaccurate, misleading, or unclear to you, 
a practicing program administrator, then it is entirely appropriate to contact 
the source of the data to politely and professionally request clarification or 
correction. In addition to addressing your immediate concerns, you may help 
the author make the data more accurate, understandable, and actionable for 
the entire education community. 


1 Forum Guide to Building a Culture of Qualitj Data: A School and District Resource, available at 
http: / / nces. ed.gov/Jorum / pub_2005801 . asp. 
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Training and Skill Development 

Current professionals might receive informal support through peer-to-peer study and 
professional learning communities. A great deal of information concerning the processes 
and techniques that support good data use can also be found online by enterprising 
individuals who understand the value of strong data use skills. More formally, professional 
development opportunities might include training that emphasize topics such as recognizing 
whether data are within acceptable ranges, identifying patterns or trends in the data, 
and applying this information to solve real-world problems. Similarly, trainers with data 
expertise can develop a staff’s quantitative skills, such as appreciating the meaning of error 
measures; calculating the amount of change a variable presented in charts, tables, and 
graphs; and interpreting the significance of statistical summaries of large data. 

A Question-Driven Approach to Data Use 

Using the skills described in the previous section, state program staff can rely on a question- 
driven approach to data use for nearly any scenario that requires making a data-informed 
decision and taking data-informed action. 

A What do I want to know? 

A What data might be relevant? 

A How will I access the relevant data? 

A What skills and tools do I need to analyze the data? 

A What do the data tell me? 

A What are my conclusions? 

A What will I do? 

A What effects did my actions have? 

A What are the next steps? 

Table 1 and table 2 illustrate how these questions can be applied to example scenarios 
commonly seen by state program staff. 
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Step 3: Step 2: Step!: Data Use 

Analyze/Interpret Data Access/Gather Data Seek Info Cycle 



Table 1. An illustration of a question-driven approach to data use for various scenarios facing state program staff 
(example 1). 


Framework 

Scenario 

Comments 

Scenario 

A state program administrator is evaluating program 

Program evaluation is a common occurrence for 


effectiveness as defined by a goal of improving the 
graduation rates of program completers. 

many state program administrators. 

What do 1 want to 

Based on this scenario, 1 need to know... 

Subsequent questions include: 

know? 



Are there enough data to draw conclusions 


Are program completers graduating from high 

about long-term trends? How are program 


school at a higher rate than their peers who do not 

completers succeeding in postsecondary 


complete the program? 

education and the workforce? 

What data might 

Based on the information 1 need, possibilities 

Although there may be a wealth of data that 

be relevant? 

include. 


could be evaluated, the scenario may warrant 
only a small subset of data for analysis. 


E 

program participation data 



E 

program completion data 



□ 

attendance data 



□ 

mobility data 



□ 

discipline data 



E 

family data 



□ 

program cost data 



□ 

academic growth data 



E 

graduation/completion data 



□ 

postsecondary education data 



□ 

workforce data 


How will 1 access 

Based on the relevant data, 1 will... 

You may already have access to these data 

the relevant 



sources: other needs may require gathering data 

data? 

□ 

consult colleagues about observational 
and anecdotal data 

you do not already have. 


□ 

institute a new collection 

The process for accessing these data may vary 


E 

review program participation and 
completion data 

by organization. 


E 

review program fiscal data 



E 

access statewide longitudinal data system 
portal 


What skills and 

Based on gaining access to these data, 1 will need to 

Don’t be too quick to form an opinion based 

tools do 1 need 

be able to... 

on one or even a few pieces of data. Use 

to analyze the 



techniques such as triangulation to assimilate and 

data? 

E 

understand the context and how the data 

evaluate the body of data in order to develop 



are presented 

conclusions. 


E 

organize the data in a way that applies to 
this situation 



E 

decide how 1 will determine student peer 
groups 



E 

relate, combine, and balance information 
from different sources 



E 

assess the data quality and limitations 


What do the data 

Based on this analysis, the data suggest... 

Is there a correlational relationship in the data? 

tell me? 

□ 

inconsistent or conflicting results 

Is there a cause-effect relationship in the data? 


E 

students who complete the program 
graduate at a higher rate 



□ 

students who complete the program 
graduate at a lower rate 



□ 

students who complete the program 
graduate at the same rate 
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Step 5: Step 4: 

Evaluate Act on the Data 



Table 1. An illustration of a question-driven approach to data use for various scenarios facing state program staff 
(continued). 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

Students who complete the program are graduating 
at a higher rate than comparable student 
populations that did not complete the program. 

If it is not a problem that requires action, then 
it is the end of the inquiry. If it is a problem that 
requires action, then more information becomes 
necessary. 

What will 1 do? 

Based on this conclusion, actions 1 will take include... 

□ find more information (check other data 
sources) 

E write a cost-benefit report 

□ recommend changes to program eligibility 

□ recommend changes to program activities 

E recommend an increase in program 

funding to increase student participation 

E communicate with: 

E eligible students 

E eligible students' parents 

E state superintendent and school board 

Can the program accommodate additional 
participation? 

Are more eligible students willing to participate? 
Will additional funding permit additional 
participation? 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 

E the program was effective with respect to 
its goal to increase graduation rates 
□ the program was not effective with respect 
to its goal to increase graduation rates 

At a later point in time, 1 can assess whether 
my action (based on data use) changed the 
situation. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

E continue to monitor the situation 

E observe whether program completers 
continue to demonstrate better 
performance than non-completers 

E evaluate whether subsequent questions 
have arisen and, if necessary, using this 
question-driven approach to data use to 
advance the inquiry 

If the situation is not completely resolved, the 
program administrator/staff may wish to identify 
what specific issues still need to be addressed 
and return to the beginning of the data use cycle 
to seek new information that will inform future 
action. 
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Table 2. An illustration of a question-driven approach to data use for various scenarios facing state program staff 
(example 2). 


Q) 

U 

> 

o 

<D 

Framework 

Scenario 

Comments 

Scenario 

A state program administrator is applying for a grant 

Grant writing is a common occurrence for many 

=> 


to improve student literacy and needs to know how 

state program administrators. 



expensive it would be to expand the program to 


Q 


new service sites. 



What do 1 want to 

Based on this scenario, 1 need to know... 

Subsequent questions include: 


know? 



How many students are eligible to participate in 

.. £ 


What is the cost of expanding the program to ten 

the program at each prospective site? 

a 


new locations? 

What is the profile of these students] (e.g., 

a «> 




academic, behavioral, social, family, health)? 

co 




What were the costs per participant at each type 
of site? 


What data might 

Based on the information 1 need, possibilities 

Although there may be a wealth of data that 


be relevant? 

include. 


could be evaluated, the scenario may warrant 
only a small subset of data for analysis. 



0 

Site location data 




0 

Program eligibility data 




□ 

Attendance data 




□ 

Mobility data 




□ 

Discipline data 




□ 

Family data 




0 

Socio-economic data 


O 


0 

Program cost data 


a 



0 Direct expenses 


.. 5 



0 Indirect expenses 


CM £ 



0 Total expenses 


(b o 

co ^ 

' «/) 



0 Cost per participant 


a> 

u 

How will 1 access 

Based on the relevant data, 1 will... 

You may already have access to these data 

< 

the relevant 



sources: other needs may require gathering data 


data? 

□ 

consult colleagues about observational 
and anecdotal data 

you do not already have. 



□ 

institute a new collection 

The process for accessing these data may vary 



0 

review grant application criteria and 
priorities 

by organization. 



0 

review program fiscal data 




0 

access GIS data on community/site need 




0 

access statewide longitudinal data system 
portal 
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Step 5: Step 4: Step 3: 

Evaluate Act on the Data Analyze/Interpret Data 



Table 2. An illustration of a question-driven approach to data use for various scenarios facing state program staff 
(continued). 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 

E understand the context and how the data 
are presented 

E organize the data in a way that applies to 
this situation 

E integrate parameters from grant 
requirements 

E relate, combine, and balance information 
from different sources 

E assess the data quality and limitations 

Don't be too quick to form an opinion based 
on one or even a few pieces of data. Use 
techniques such as triangulation to assimilate and 
evaluate the body of data in order to develop 
conclusions. 

What do the data 
tell me? 

Based on this analysis, the data suggest... 

□ inconsistent or conflicting results 

E there is a need and corresponding 
opportunity for expansion 

□ there is not a need for expansion 

□ there is not an opportunity for expansion 

Is there a correlational relationship in the data? 

Is there a cause-effect relationship in the data? 

What are my 
conclusions? 

Based on this interpretation, 1 conclude... 

Our agency should apply for the funds needed to 
serve additional sites that have students who are 
eligible to receive service but not currently enrolled. 

If it is not a problem that requires action, then 
it is the end of the inquiry. If it is a problem that 
requires action, then more information becomes 
necessary. 

What will 1 do? 

Based on this conclusion, actions 1 will take include... 

□ find more information (check other data 
sources) 

□ do not apply for a grant 

E recommend writing a grant application 

□ recommend changes to program eligibility 

E communicate with: 

□ eligible students 

□ eligible students' parents 

E state superintendent and school board 

Steps must be taken to actively apply for the 
grant if that is the action that is warranted by the 
information. 

What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 

□ the grant application was not submitted 

E the grant application was submitted 

At a later point in time, 1 can assess whether 
my action (based on data use) changed the 
situation. 

What are the 
next steps? 

Based on my evaluation, 1 will... 

E continue to monitor the situation 

E observe whether the grant application was 
successful and funds were received to 
increase the number of students served 

E if the grant was not received, evaluate 
whether subsequent or alternative 
opportunities for funding should be 
explored, using this question-driven 
approach to data use to advance the 
inquiry 

If the situation is not completely resolved, the 
program administrator/staff may wish to identify 
what specific issues still need to be addressed 
and return to the beginning of the data use cycle 
to seek new information that will inform future 
action. 
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Conclusions 

High quality and timely data are powerful tools for improving the management and 
operation of the entire education enterprise. Thoughtful application of the frameworks, 
principles, and questions recommended in this Brief will help state program staff use data 
to inform decisionmaking and actions related to program services and management. The 
Practical Exercise below provides resources for applying these recommendations when 
dealing with actual issues or illustrative scenarios. While these skills can be developed 
by individuals, skill development at a staff level will be enhanced by the delivery of 
professional development curricula around this area of growing importance for the entire 
education enterprise. Many state program staff will benefit by collaborating with colleagues 
when completing these data use scenarios. 
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1 A 


Practical Exercise for State Program Staff: Using What You've Learned 

State program staff face a wide range of issues that require that they make decisions each and every day. Following the 
question-driven approach to data use described above, work individually or brainstorm with colleagues to complete 
the tables below. Be as realistic as possible in your responses, using real data sources, practices, and people in your 
organization so as to make the scenario as useful as possible. 



Framework 

Scenario 

Comments 

Scenario 

A state program staff person is preparing a budget 
for the upcoming service year. 

A common occurrence for state program staff. 

What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the data 
tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Step 5: Step 4: Step 3: Step 2: Stepl: Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 


4fc* 9)1 


Scenario 2 


Framework 

Scenario 

Comments 

Scenario 

A state program staff person is assessing the 
effectiveness of program staff. 

A common occurrence for state program staff. 

What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the data 
tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Step 5: Step 4: Step 3: Step 2: Step 1 : Data Use 

Evaluate Act on the Data Analyze/Interpret Data Access/Gather Data Seek Info Cycle 



Insert Your Own Scenario Table 


Framework 

Scenario 

Comments 

Scenario 



What do 1 want to 
know? 

Based on this scenario, 1 need to know... 


What data might 
be relevant? 

Based on the information 1 need, possibilities 
include... 


How will 1 access 
the relevant 
data? 

Based on the relevant data, 1 will... 


What skills and 
tools do 1 need 
to analyze the 
data? 

Based on gaining access to these data, 1 will need to 
be able to... 


What do the data 
tell me? 

Based on this analysis, the data suggest... 


What are my 
conclusions? 

Based on this interpretation, 1 conclude... 


What will 1 do? 

Based on this conclusion, actions 1 will take include... 


What effects 
did my actions 
have? 

Based on these actions, 1 observed that... 


What are the 
next steps? 

Based on my evaluation, 1 will... 
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Appendix: Online Resources 

The following resources are available online and may serve as useful tools or references for 
learning more about good data use. 

Achieving a Wealth of Riches: Delivering on the Promise of Data to Transform 
Teaching and Learning 

http: / / www.all4ed.org/Jiles/AchievingWealth0jKiches.pdf 
Author: Miller (2009) 

Description: This brief addresses why using data represents a significant shift for most 
teachers in how they perform their jobs, explains the importance of using multiple types 
of data to affect learning, details the infrastructure necessary to encourage teachers 1 use of 
data, and provides federal policy recommendations. 

Benefits of and Lessons Learned from Linking Teacher and Student Data 

http: / / www.dataqualitycampaign.org/jles/publications-benefits_oj_and_lessons_learned_from_ 
linking_teacher_and_student_data- 1 20601 .pdf 
Author: DQC (2007) 

Description: This brief looks at how several states are linking student and teacher data, the 
benefits of those links, and lessons learned. 

Can interim assessments be used for instructional change? 

http: / / www.cpre.org/images/stories/ cpre_pdjs / rb_5 1 _role%20policy%20briej_final%20web.pdj 
Author: Goertz, et al. (2009) 

Description: The purpose of this study was to explore the use of interim assessments in 
elementary mathematics and the district and school policies that support the use of interim 
assessments to modify classroom instruction. This use of interim assessments was supported 
by various district and school factors including the alignment of assessment content to 
standards and curriculum, district expectations that interim assessments should be used 
to inform instruction, a high-quality and accessible information management system, time 
allocated for re-teaching, instructional supports for students, and professional supports 
for teachers. The researchers conclude with a discussion of the implications for policy and 
research. 
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Continuous Improvement: It Takes More Than Test Scores 

http: / / eff.csuchico. edu/ downloads / TestScores. pelf 
Author: Bernhardt (2004) 

Description: This article summarizes why analyzing state assessment results is only the 
beginning of effective data-driven decisionmaking. 

A critical review of research on formative assessments: The limited scientific 
evidence of the impact of formative assessments in education 

http: / / pareonline.net / pdf / vl4n7.pdf 
Author: Dunn & Mulvenon (2009) 

Description: The purpose of this article is to review the research on formative assessment. 
The authors clarify terms related to formative assessment practices and then provide a 
critical review of the formative assessment literature . They conclude with recommendations 
concerning the use of a common vocabulary when researching formative assessment 
practices as well as additional research on the effects of formative assessments on student 
performance. 

Cutting through the “data-driven” mantra: Different conceptions of data- 
driven decision making. 

http: / / www.rand.org/pubs/reprints/ 2009 / RAND_RP 1 37 2. pelf 
Author: Ikemoto, G.S. & Marsh, J.A. (2007) 

Description: This chapter explores the increased emphasis on data-driven decisionmaking 
(DDDM) and the ways in which educators use data to make informed decisions about 
teaching and learning. The authors explain the process of DDDM from data collection to 
implementation and identify four broad models of DDDM strategies that range from simple 
to complex. 

Data for School Improvement: Factors for designing effective information 
systems to support decision-making in schools 

http:/ / www.iJets.info/journals/9_3/ lS.pef 7 
Author: Breiter, Light (2006) 

Description: This document draws on the considerable body of business and organizational 
research on MIS (and a recent educational case study in New York City) to introduce a 
theoretical framework for getting from data to decisionmaking in schools. It explores how 
schools use information with an emphasis on the potential of new technologies and new 
ways of analysis to meet the information needs of educators across different levels of the 
system. 
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Data use in the school and classroom: The challenges of implementing data- 
based decision making inside schools 

http: / / www.wcer.wisc.edu/puhlications/ workingPapers / Working_Paper_No_2002_2 .pdf 
Author: Thorn, C.A. (2002) 

Description: This paper explores the challenges teachers and school-level administrators 
face when implementing research-based decisions about instructional practices in the 
classroom. The author presents both individual and group strategic decision making models 
and maintains that no one model is best. The author concludes with recommendations 
to improve quality decisionmaking, including greater access to data and high quality 
professional development opportunities, incentives for short- and long-term gains, and an 
expanded range of strategies to address plans for school improvement. 

How to Read a Research Article 

http: / / www.human.cornell.edu/pam/ outreach /parenting /research/ upload/How-20to-20Read- 

20a-20Research-20Article.pdf 

Author: Dunifon (2005) 

Description: This brief describes a step-by-step process for making reading a research 
article more illuminating and useful to non-researchers. 

Making Sense of All Your Data 

http: / / www.principals.org / portals / 0 / content / 54342.pdf 
Author: LaChat, et al. (2006) 

Description: This article discusses the three key components of effective data use: essential 
questions that guide data analysis, data warehousing technology that supports extensive data 
disaggregation, and a data team and data coach that ensures that data are available and used. 
These components support a culture of inquiry, continuous improvement, accountability, 
and purposeful data-driven decisionmaking as cornerstones to ensure student success. 

Making Sense of Data-Driven Decision Making in Education: Evidence from 
Recent RAND Research 

http: / / www.rand.org/content/ dam / rand / pubs / occasional_papers / 2006 / RAND_OP 1 70.pdf 
Author: RAND (2006) 

Description: Data-driven decisionmaking applied to student achievement testing data is a 
central focus of many school and district reform efforts, in part because of federal and state 
test-based accountability policies. This paper shows how schools and districts are analyzing 
achievement test results and other types of data to make decisions to improve student 
success. 

Multiple Measures 

http: / / eff.csuchico. edu/ downloads / MMeasure.pdf 
Author: Bernhardt (1998) 

Description: This article advocates for the use of a variety of data to measure student 
learning and school success. In addition to assessment data, the author argues for the use of 
demographics, perceptions, and school process data. 
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The Next Step: Using Longitudinal Data Systems to Improve Student Success 

http: / / www.dataqualitycampaign.org/jiles/NextStep.pdf 
Author: DQC (2009) 

Description: This report looks at the “next step” of the LDS development movement: using 
data to improve education. It summarizes a number of key tasks that education agencies 
could undertake to facilitate effective data use (expand linkages; ensure access to, analysis 
of, and communication about data; and build skills among stakeholders to effectively use 
data) . 

A Policymaker’s Guide to the Value of Student Longitudinal Data 

http: / / www. ecs. org / clearinghouse 140/211 402 1 .htm 
Author: Dougherty, C (2002) 

Description: This report on the use of longitudinal data addresses the types of policy 
questions an LDS data can help to answer. 

Practices That Support Data Use 

http:/ / daniellight.edublogs.org /files/ 2007 / 1 1 / lachat-smith-2005 .pdf 
Author: Lachat, Smith (2005) 

Description: This article presents initial findings of a case study focused on data use in five 
low-performing urban high schools undergoing comprehensive schoolwide reform. The 
findings point to several key factors that have an impact on data use in the study sites: 
the quality and accuracy of available data, staff access to timely data, the capacity for data 
disaggregation, the collaborative use of data organized around a clear set of questions, and 
leadership structures that support school-wide use of data. 

Reporting and Analysis Tools 

http: / / www. dataqualitycampaign. org /files/ Publications-Reporting_and_Analysis_Tools_ 
Education_Data-090 1 07 .pdf 
Author: DQC (2007) 

Description: This policy brief explores how to make data useful and accessible, and reviews 
lessons learned from states implementing reporting and analysis tools. 

Tapping into the Power of Longitudinal Data: A Guide for School Leaders 

http: / / dataqualitycampaign.org/files/publications-tapping_into_the_power_oJ_longitudinal_ 
data-a _guide_for_school_leaders-01 01 08.pdf 
Author: DQC (2008) 

Description: In this guide for school leaders, the DQC looks at ways teachers and principals 
can use longitudinal data to meet students’ individual needs and improve performance. 
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Teachers’ Use of Data to Improve Student Learning 

http: / / www.docstoc.com/ docs/ 436877 26 / Teachers-Use-of-Data-to-Improve-Student-Learning 
Author: Urbanski (2004) 

Description: This article focuses on the value of using data to improve teaching and 
learning, with descriptions of the types of resources needed to prepare teachers to use data 
effectively. 

Teachers’ Use of Student Data Systems to Improve Instruction: 2005 to 2007 

http: / / www2 .ed.gov / rschstat / eval / tech! teachers-data-use-2005-2007 / teachers -data- 

use-2005-2007.pdf 

Author: Gallagher, et al. (2008) 

Description: The availability and use of electronic student data systems for instructional 
improvement are rapidly changing. Federal, state, and district efforts to promote the use of 
data to improve instruction and student achievement are being accompanied by changes in 
teacher access to student data systems and data use. This report summarizes survey results 
on teacher use of data to support their work. 

Teachers’ Use of Student Level Data Systems 

http: / / www.ed.gov/rschstat/eval/ tech/ teachers-data-use / teachers-data-use-intro.html 
Author: USED (2007) 

Description: Relying on data from national surveys of teachers and school districts, this 
brief provides the first national estimates of the prevalence of K-12 teacher access to and 
use of electronic student data management systems. 

What’s the evidence on districts’ use of evidence? 

http: / / www-gse.herkeley.edu/Jaculty/ CECoburn/coburnhonigsteinJnal.pcJ 
Author: Coburn, et al. (2009) 

Description: This report documents findings from an extensive review of research on 
school districts’ use of evidence in decisionmaking. The authors discover that the processes 
by which district personnel use evidence to inform their decisions are often complex 
and mediated by individual and communal interpretations of the research, as well as by 
organizational and political conditions. Based on these findings the authors provide a 
set of key lessons to support evidence use in districts and highlight the importance of 
understanding the processes at work. 
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